{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# XGBoost Parameter Tuning for Happy Customer Bank"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 调整树的参数：max_depth & min_child_weight\n",
    "(粗调，参数的步长为2；下一步是在粗调最佳参数周围，将步长降为1，进行精细调整)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "数据说明:\n",
    "数据集共26个字段: 其中1-24列为输入特征，25-26列为输出特征。\n",
    "    1. ID - 唯一ID（不能用于预测）\n",
    "    2. Gender - 性别\n",
    "    3. City - 城市\n",
    "    4. Monthly_Income - 月收入（以卢比为单位）\n",
    "    5. DOB - 出生日期\n",
    "    6. Lead_Creation_Date - 潜在（贷款）创建日期\n",
    "    7. Loan_Amount_Applied - 贷款申请请求金额（印度卢比，INR）\n",
    "    8. Loan_Tenure_Applied - 贷款申请期限（单位为年）\n",
    "    9. Existing_EMI -现有贷款的EMI（EMI：电子货币机构许可证） \n",
    "    10. Employer_Name雇主名称\n",
    "    11. Salary_Account - 薪资帐户银行\n",
    "    12. Mobile_Verified - 是否移动验证（Y / N）\n",
    "    13. VAR5 - 连续型变量\n",
    "    14. VAR1-  类别型变量\n",
    "    15. Loan_Amount_Submitted - 提交的贷款金额（在看到资格后修改和选择）\n",
    "    16. Loan_Tenure_Submitted - 提交的贷款期限（单位为年，在看到资格后修改和选择）\n",
    "    17. Interest_Rate - 提交贷款金额的利率\n",
    "    18. Processing_Fee - 提交贷款的处理费（INR）\n",
    "    19. EMI_Loan_Submitted -提交的EMI贷款金额（INR）\n",
    "    20. Filled_Form - 后期报价后是否已填写申请表格\n",
    "    21. Device_Type - 进行申请的设备（浏览器/移动设备）\n",
    "    22. Var2 - 类别型变量\n",
    "    23. Source - 类别型变量\n",
    "    24. Var4 - 类别型变量\n",
    "\n",
    "输出：\n",
    "    25. LoggedIn - 是否login（只用于理解问题的变量，不能用于预测，测试集中没有）\n",
    "    26. Disbursed - 是否发放贷款（目标变量），1为发放贷款（目标客户）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from xgboost import XGBClassifier\n",
    "import xgboost as xgb\n",
    "\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "from sklearn.metrics import log_loss\n",
    "\n",
    "from matplotlib import pyplot\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = pd.read_csv('train_save.csv')\n",
    "test = pd.read_csv('test_save.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((87020, 52), (37717, 51))"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.shape, test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAEKCAYAAADaa8itAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAFpFJREFUeJzt3Xu0JWV95vHvw02Qi4C2LtNAGmNHJTreWoImEy8oAk5oZEGCyxEGmcFkAHESHTHJsuNdEoRI4iWsQLiMw0UGtEWUQQSdOIo0F8UGWXQwQg8EWkFEGGEafvPHfg/s7j6Xorv2OWzP97PWXqfqrbdq/7ar5VlVb9VbqSokSerDZnNdgCTpV4ehIknqjaEiSeqNoSJJ6o2hIknqjaEiSeqNoSJJ6o2hIknqjaEiSerNFnNdwGx7xjOeUYsWLZrrMiRpbFxzzTU/qaoFXfrOu1BZtGgRK1asmOsyJGlsJPlx175e/pIk9cZQkST1xlCRJPXGUJEk9cZQkST1xlCRJPXGUJEk9cZQkST1xlCRJPVm3j1Rv6le/p6z5roEPQld89eHzXUJ0pOCZyqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTejDRUkvyXJCuT/CDJOUm2TrJ7kquS3JLkvCRbtb5Paeur2vZFQ8d5X2u/Ockbh9r3bW2rkhw/yt8iSZrZyEIlyULgncCSqnohsDlwKHACcHJVLQbuBY5suxwJ3FtVzwVObv1Iskfb77eAfYFPJ9k8yebAp4D9gD2At7S+kqQ5MurLX1sA2yTZAngqcCfwOuCCtv1M4MC2vLSt07bvnSSt/dyqeqiqfgSsAvZsn1VVdWtVPQyc2/pKkubIyEKlqv4PcCJwG4MwuQ+4BvhZVa1t3VYDC9vyQuD2tu/a1v/pw+3r7TNVuyRpjozy8tdODM4cdgd+DdiWwaWq9dXELlNse6Ltk9VyVJIVSVasWbNmptIlSRtplJe/Xg/8qKrWVNX/Ay4EXgXs2C6HAewC3NGWVwO7ArTtTwPuGW5fb5+p2jdQVadW1ZKqWrJgwYI+fpskaRKjDJXbgL2SPLWNjewN3AhcARzc+hwOfLEtL2/rtO1fr6pq7Ye2u8N2BxYD3wWuBha3u8m2YjCYv3yEv0eSNIORvaSrqq5KcgFwLbAWuA44FfgycG6SD7e209oupwFnJ1nF4Azl0HaclUnOZxBIa4Gjq+oRgCTHAJcyuLPs9KpaOarfI0ma2Ujf/FhVy4Bl6zXfyuDOrfX7/hI4ZIrjfAT4yCTtlwCXbHqlkqQ++ES9JKk3hookqTeGiiSpN4aKJKk3hookqTeGiiSpN4aKJKk3hookqTeGiiSpN4aKJKk3hookqTeGiiSpN4aKJKk3hookqTeGiiSpN4aKJKk3hookqTeGiiSpN4aKJKk3hookqTeGiiSpN4aKJKk3M4ZKkm2TbNaWfzPJAUm2HH1pkqRx0+VM5ZvA1kkWApcDRwBnjLIoSdJ46hIqqaoHgYOAv62qNwN7jLYsSdI46hQqSV4JvBX4cmvbYnQlSZLGVZdQeRfwPuCiqlqZ5DnAFaMtS5I0jmY846iqbwDfSLJtW78VeOeoC5MkjZ8ud3+9MsmNwE1t/cVJPj3yyiRJY6fL5a+/Ad4I/BSgqr4H/N4oi5IkjadODz9W1e3rNT0yglokSWOuy11ctyd5FVBJtmIwnnLTaMuSJI2jLmcqfwQcDSwEVgMvaeuSJK2jy91fP2HwjIokSdPqcvfXmUl2HFrfKcnpoy1LkjSOulz++jdV9bOJlaq6F3jp6EqSJI2rLqGyWZKdJlaS7IzTtEiSJtElHD4B/O8kF7T1Q4CPjK4kSdK46jJQf1aSa4DXAgEOqqobR16ZJGnsdH3z4w+BC4EvAr9IsluXnZLsmOSCJD9MclOb8mXnJJcluaX93an1TZJTkqxK8v0kLxs6zuGt/y1JDh9qf3mSG9o+pyRJ958uSepbl7u/jgXuAi4DLmYw/f3FHY//SeCrVfV84MUMHpo8Hri8qhYzeOnX8a3vfsDi9jkK+Ez7/p2BZcBvA3sCy4bGeD7T+k7st2/HuiRJI9BlTOU44HlV9dMncuAkOzCYI+w/AFTVw8DDSZYCr2ndzgSuBN4LLAXOqqoCvtPOcp7d+l5WVfe0414G7JvkSmCHqvp2az8LOBD4yhOpU5LUny6Xv24H7tuIYz8HWAP8Y5LrkvxDmz7/WVV1J0D7+8zWf2H7rgmrW9t07asnaZckzZEuZyq3Alcm+TLw0ERjVZ3U4dgvA46tqquSfJLHL3VNZrLxkNqI9g0PnBzF4DIZu+3WaThIkrQRupyp3MZgPGUrYPuhz0xWA6ur6qq2fgGDkLmrXdai/b17qP+uQ/vvAtwxQ/suk7RvoKpOraolVbVkwYIFHUqXJG2MLrcUfwAgybZV9UDXA1fVvya5PcnzqupmYG/gxvY5HPh4+/vFtsty4Jgk5zIYlL+vqu5Mcinw0aHB+X2A91XVPUnuT7IXcBVwGPC3XeuTJPVvxlBJ8krgNGA7YLckLwbeUVX/ucPxjwU+16bMvxU4gsHZ0flJjmRwFnRI63sJsD+wCniw9aWFx4eAq1u/D04M2gN/DJwBbMNggN5BekmaQ13GVCbe/LgcBm9+TNLpzY9VdT2wZJJNe0/St5hiSv2qOh3YYBLLqloBvLBLLZKk0fPNj5Kk3vjmR0lSb3zzoySpN9OeqSTZHHhbVfnmR0nSjKY9U6mqRxhMnyJJ0oy6jKl8K8nfAecBjz2nUlXXjqwqSdJY6hIqr2p/PzjUVsDr+i9HkjTOZhpT2Qz4TFWdP0v1SJLG2ExjKo8Cx8xSLZKkMdflluLLkrw7ya7trY07txdnSZK0ji5jKm9vf4efTSkG70uRJOkxXWYp3n02CpEkjb8usxQfNll7VZ3VfzmSpHHW5fLXK4aWt2Yww/C1gKEiSVpHl8tfxw6vJ3kacPbIKpIkja1OU9+v50Fgcd+FSJLGX5cxlS8xuNsLBiG0B+DDkJKkDXQZUzlxaHkt8OOqWj2ieiRJY6xLqNwG3FlVvwRIsk2SRVX1LyOtTJI0drqMqXweeHRo/ZHWJknSOrqEyhZV9fDESlveanQlSZLGVZdQWZPkgImVJEuBn4yuJEnSuOoypvJHwOfai7pg8J76SZ+ylyTNb10efvxnYK8k2wGpqvtHX5YkaRzNePkryUeT7FhVv6iq+5PslOTDs1GcJGm8dBlT2a+qfjaxUlX3AvuPriRJ0rjqEiqbJ3nKxEqSbYCnTNNfkjRPdRmo/2/A5Un+kcF0LW8HzhxpVZKksdRloP6vknwfeH1r+lBVXTrasiRJ46jLmQrAdcCWDM5UrhtdOZKkcdbl7q8/AL4LHAz8AXBVkoNHXZgkafx0OVP5c+AVVXU3QJIFwNeAC0ZZmCRp/HS5+2uziUBpftpxP0nSPNPlTOWrSS4FzmnrfwhcMrqSJEnjqsvdX+9JchDwu0CAU6vqopFXJkkaO53u/qqqC4ELR1yLJGnMOTYiSeqNoSJJ6s2UoZLk8vb3hNkrR5I0zqYbU3l2klcDByQ5l8Eg/WOq6tqRViZJGjvTXf56P3A8sAtwEvCJoc+JXb8gyeZJrktycVvfPclVSW5Jcl6SrVr7U9r6qrZ90dAx3tfab07yxqH2fVvbqiTHd//ZkqRRmDJUquqCqtoP+Kuqeu16n9c9ge84DrhpaP0E4OSqWgzcCxzZ2o8E7q2q5wInt34k2QM4FPgtYF/g0y2oNgc+BewH7AG8pfWVJM2RGQfqq+pDSQ5IcmL7/LuuB0+yC/Am4B/aeoDX8fgUL2cCB7blpTw+pf4FwN6t/1Lg3Kp6qKp+BKwC9myfVVV1a1U9DJzb+kqS5kiXCSU/xuBs48b2Oa61dfE3wH8FHm3rTwd+VlVr2/pqYGFbXgjcDtC239f6P9a+3j5TtU/2G45KsiLJijVr1nQsXZL0RHW5pfhNwBuq6vSqOp3BJag3zbRTO6O5u6quGW6epGvNsO2Jtm/YWHVqVS2pqiULFiyYpmpJ0qbo+j6VHYF72vLTOu7zOwzuHNsf2BrYgcGZy45JtmhnI7sAd7T+q4FdgdVJtmjfc89Q+4ThfaZqlyTNgS5nKh8DrktyRpIzgWuAj860U1W9r6p2qapFDAbav15VbwWuYPBuFoDDgS+25eVtnbb961VVrf3QdnfY7sBiBu93uRpY3O4m26p9x/IOv0eSNCJdJpQ8J8mVwCsYXHJ6b1X96yZ853uBc5N8mMFbJE9r7acBZydZxeAM5dD2/SuTnM9gPGctcHRVPQKQ5BjgUmBz4PSqWrkJdUmSNlHXCSXvZBPOAqrqSuDKtnwrgzu31u/zS+CQKfb/CPCRSdovwWn4JelJw7m/JEm9MVQkSb2ZNlSSbJbkB7NVjCRpvE0bKlX1KPC9JLvNUj2SpDHWZaD+2cDKJN8FHphorKoDRlaVJGksdQmVD4y8CknSr4Quz6l8I8mvA4ur6mtJnsrguRBJktbRZULJ/8Rg1uC/b00LgS+MsihJ0njqckvx0Qzm8fo5QFXdAjxzlEVJksZTl1B5qL2vBIA22eOkswFLkua3LqHyjSR/BmyT5A3A54EvjbYsSdI46hIqxwNrgBuAdzCYa+svRlmUJGk8dbn769E25f1VDC573dympJckaR0zhkqSNwGfBf6ZwdT3uyd5R1V9ZdTFSZLGS5eHHz8BvLaqVgEk+Q3gy4ChIklaR5cxlbsnAqW5Fbh7RPVIksbYlGcqSQ5qiyuTXAKcz2BM5RAGr/KVJGkd013++v2h5buAV7flNcBOI6tIkjS2pgyVqjpiNguRJI2/Lnd/7Q4cCywa7u/U95Kk9XW5++sLwGkMnqJ/dLTlSJLGWZdQ+WVVnTLySiRJY69LqHwyyTLgfwIPTTRW1bUjq0qSNJa6hMqLgLcBr+Pxy1/V1iVJekyXUHkz8Jzh6e8lSZpMlyfqvwfsOOpCJEnjr8uZyrOAHya5mnXHVLylWJK0ji6hsmzkVUiSfiV0eZ/KN2ajEEnS+OvyRP39PP5O+q2ALYEHqmqHURYmSRo/Xc5Uth9eT3IgsOfIKpIkja0ud3+to6q+gM+oSJIm0eXy10FDq5sBS3j8cpgkSY/pcvfX8HtV1gL/AiwdSTWSpLHWZUzF96pIkjqZ7nXC759mv6qqD42gHknSGJvuTOWBSdq2BY4Eng4YKpKkdUz3OuFPTCwn2R44DjgCOBf4xFT7SZLmr2nHVJLsDPwJ8FbgTOBlVXXvbBQmSRo/Uz6nkuSvgauB+4EXVdVfPpFASbJrkiuS3JRkZZLjWvvOSS5Lckv7u1NrT5JTkqxK8v0kLxs61uGt/y1JDh9qf3mSG9o+pyTJRvxvIEnqyXQPP/4p8GvAXwB3JPl5+9yf5Ocdjr0W+NOqegGwF3B0kj2A44HLq2oxcHlbB9gPWNw+RwGfgcfOlpYBv83gSf5lE0HU+hw1tN++3X62JGkUpgyVqtqsqrapqu2raoehz/Zd5v2qqjsnXjlcVfcDNwELGTzjcmbrdiZwYFteCpxVA98BdkzybOCNwGVVdU87U7oM2Ldt26Gqvl1VBZw1dCxJ0hx4wtO0bIwki4CXAlcBz6qqO2EQPMAzW7eFwO1Du61ubdO1r56kfbLvPyrJiiQr1qxZs6k/R5I0hZGHSpLtgP8BvKuqprtsNtl4SG1E+4aNVadW1ZKqWrJgwYKZSpYkbaSRhkqSLRkEyueq6sLWfFe7dEX7e3drXw3sOrT7LsAdM7TvMkm7JGmOjCxU2p1YpwE3VdVJQ5uWAxN3cB0OfHGo/bB2F9hewH3t8tilwD5JdmoD9PsAl7Zt9yfZq33XYUPHkiTNgS4TSm6s3wHeBtyQ5PrW9mfAx4HzkxwJ3AYc0rZdAuwPrAIeZPCgJVV1T5IPMbi9GeCDVXVPW/5j4AxgG+Ar7SNJmiMjC5Wq+icmH/cA2HuS/gUcPcWxTgdOn6R9BfDCTShTktSjWbn7S5I0PxgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN6Mfagk2TfJzUlWJTl+ruuRpPlsi7kuYFMk2Rz4FPAGYDVwdZLlVXXj3FYmzY3bPviiuS5BT0K7vf+GWfuucT9T2RNYVVW3VtXDwLnA0jmuSZLmrXEPlYXA7UPrq1ubJGkOjPXlLyCTtNUGnZKjgKPa6i+S3DzSquaPZwA/mesingxy4uFzXYI25L/PCcsm+0/lE/LrXTuOe6isBnYdWt8FuGP9TlV1KnDqbBU1XyRZUVVL5roOaTL++5wb437562pgcZLdk2wFHAosn+OaJGneGuszlapam+QY4FJgc+D0qlo5x2VJ0rw11qECUFWXAJfMdR3zlJcU9WTmv885kKoNxrUlSdoo4z6mIkl6EjFUNKOZpsJJ8pQk57XtVyVZNPtVaj5KcnqSu5P8YIrtSXJK+7f5/SQvm+0a5xtDRdMamgpnP2AP4C1J9liv25HAvVX1XOBk4ITZrVLz2BnAvtNs3w9Y3D5HAZ+ZhZrmNUNFM+kyFc5S4My2fAGwd5JNftpKmklVfRO4Z5ouS4GzauA7wI5Jnj071c1Phopm0mUqnMf6VNVa4D7g6bNSnTQ9p3KaZYaKZtJlKpxO0+VIc8B/m7PMUNFMukyF81ifJFsAT2P6SxLSbOk0lZP6Y6hoJl2mwlkOTMyoeDDw9fIBKD05LAcOa3eB7QXcV1V3znVRv8rG/ol6jdZUU+Ek+SCwoqqWA6cBZydZxeAM5dC5q1jzSZJzgNcAz0iyGlgGbAlQVZ9lMNvG/sAq4EHgiLmpdP7wiXpJUm+8/CVJ6o2hIknqjaEiSeqNoSJJ6o2hIknqjaEidZTkkSTXJ1mZ5HtJ/iTJZm3bkiSnTLPva5JcPHvVbvD9f5nk3XP1/Zo/fE5F6u7/VtVLAJI8E/jvDGYPWFZVK4AVo/riJFu0edWkJzXPVKSNUFV3M5hK/Zj2tPZjZyJJXt3OaK5Pcl2S7dtuOyS5KMmNST47dJbzi4njJjk4yRlt+YwkJyW5AjhhquMmeU+Sq9v7Qj4wdKw/b+/B+RrwvNn430XyTEXaSFV1awuGZ6636d3A0VX1rSTbAb9s7XsyeCfNj4GvAgcxeFXAdH4TeH1VPZLkS+sfN8k+DN4VsieDyROXJ/k94AEGMxu8lMH/z68Frtm0XyzNzDMVadNMNgvut4CTkrwT2HHostV323tpHgHOAX63w/E/3/pPddx92uc6BsHxfAYh82+Bi6rqwar6ORvO1yaNhKEibaQkzwEeAe4ebq+qjwP/EdgG+E6S509sWu8QNUn71uv1eWCG4wb4WFW9pH2eW1WnTfF90sgZKtJGSLIA+Czwd+vPyJzkN6rqhqo6gcHg/USo7Nlme94M+EPgn1r7XUle0NrfPM13TnbcS4G3t8thJFnYbiL4JvDmJNu0sZff7+u3S9NxTEXqbpsk1zOYBXctcDZw0iT93pXktQzOYm4EvgK8Evg28HHgRQz+o39R6388cDGDNxT+ANhuiu/f4LhV9VCSFwDfbm9w/gXw76vq2iTnAdczGMP5X5vyw6WunKVYktQbL39JknpjqEiSemOoSJJ6Y6hIknpjqEiSemOoSJJ6Y6hIknpjqEiSevP/AQfaa5oYJqxDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.countplot(train.Disbursed);\n",
    "pyplot.xlabel('Disbursed');\n",
    "pyplot.ylabel('Number of occurrences');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "样本不均衡，使用交叉验证"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare cross validation\n",
    "kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "默认参数，此时学习率为0.1，比较大，观察弱分类数目的大致范围\n",
    "（采用默认参数配置，看看模型是过拟合还是欠拟合）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = train['Disbursed']\n",
    "train = train.drop([\"Disbursed\"], axis=1)\n",
    "X_train = np.array(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 87020 entries, 0 to 87019\n",
      "Data columns (total 51 columns):\n",
      "Existing_EMI                     87020 non-null float64\n",
      "Loan_Amount_Applied              87020 non-null float64\n",
      "Loan_Tenure_Applied              87020 non-null float64\n",
      "Monthly_Income                   87020 non-null int64\n",
      "Var4                             87020 non-null int64\n",
      "Var5                             87020 non-null int64\n",
      "Age                              87020 non-null int64\n",
      "EMI_Loan_Submitted_Missing       87020 non-null int64\n",
      "Interest_Rate_Missing            87020 non-null int64\n",
      "Loan_Amount_Submitted_Missing    87020 non-null int64\n",
      "Loan_Tenure_Submitted_Missing    87020 non-null int64\n",
      "Processing_Fee_Missing           87020 non-null int64\n",
      "Device_Type_0                    87020 non-null int64\n",
      "Device_Type_1                    87020 non-null int64\n",
      "Filled_Form_0                    87020 non-null int64\n",
      "Filled_Form_1                    87020 non-null int64\n",
      "Gender_0                         87020 non-null int64\n",
      "Gender_1                         87020 non-null int64\n",
      "Var1_0                           87020 non-null int64\n",
      "Var1_1                           87020 non-null int64\n",
      "Var1_2                           87020 non-null int64\n",
      "Var1_3                           87020 non-null int64\n",
      "Var1_4                           87020 non-null int64\n",
      "Var1_5                           87020 non-null int64\n",
      "Var1_6                           87020 non-null int64\n",
      "Var1_7                           87020 non-null int64\n",
      "Var1_8                           87020 non-null int64\n",
      "Var1_9                           87020 non-null int64\n",
      "Var1_10                          87020 non-null int64\n",
      "Var1_11                          87020 non-null int64\n",
      "Var1_12                          87020 non-null int64\n",
      "Var1_13                          87020 non-null int64\n",
      "Var1_14                          87020 non-null int64\n",
      "Var1_15                          87020 non-null int64\n",
      "Var1_16                          87020 non-null int64\n",
      "Var1_17                          87020 non-null int64\n",
      "Var1_18                          87020 non-null int64\n",
      "Var2_0                           87020 non-null int64\n",
      "Var2_1                           87020 non-null int64\n",
      "Var2_2                           87020 non-null int64\n",
      "Var2_3                           87020 non-null int64\n",
      "Var2_4                           87020 non-null int64\n",
      "Var2_5                           87020 non-null int64\n",
      "Var2_6                           87020 non-null int64\n",
      "Mobile_Verified_0                87020 non-null int64\n",
      "Mobile_Verified_1                87020 non-null int64\n",
      "Source_0                         87020 non-null int64\n",
      "Source_1                         87020 non-null int64\n",
      "Source_2                         87020 non-null int64\n",
      "Source_3                         87020 non-null int64\n",
      "Source_4                         87020 non-null int64\n",
      "dtypes: float64(3), int64(48)\n",
      "memory usage: 33.9 MB\n"
     ]
    }
   ],
   "source": [
    "train.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "第一轮参数调整得到的n_estimators最优值（93），其余参数继续默认值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'max_depth': [3, 5, 7, 9], 'min_child_weight': [1, 3, 5]}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#max_depth 建议3-10， min_child_weight=1／sqrt(ratio_rare_event) =5.5\n",
    "max_depth = range(3,10,2)\n",
    "min_child_weight = range(1,6,2)\n",
    "param_test2_1 = dict(max_depth=max_depth, min_child_weight=min_child_weight)\n",
    "param_test2_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ice2018/anaconda2/lib/python2.7/site-packages/sklearn/model_selection/_search.py:762: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: -0.06455, std: 0.00100, params: {'max_depth': 3, 'min_child_weight': 1},\n",
       "  mean: -0.06456, std: 0.00108, params: {'max_depth': 3, 'min_child_weight': 3},\n",
       "  mean: -0.06456, std: 0.00118, params: {'max_depth': 3, 'min_child_weight': 5},\n",
       "  mean: -0.06404, std: 0.00119, params: {'max_depth': 5, 'min_child_weight': 1},\n",
       "  mean: -0.06411, std: 0.00110, params: {'max_depth': 5, 'min_child_weight': 3},\n",
       "  mean: -0.06413, std: 0.00106, params: {'max_depth': 5, 'min_child_weight': 5},\n",
       "  mean: -0.06418, std: 0.00110, params: {'max_depth': 7, 'min_child_weight': 1},\n",
       "  mean: -0.06439, std: 0.00139, params: {'max_depth': 7, 'min_child_weight': 3},\n",
       "  mean: -0.06414, std: 0.00136, params: {'max_depth': 7, 'min_child_weight': 5},\n",
       "  mean: -0.06498, std: 0.00116, params: {'max_depth': 9, 'min_child_weight': 1},\n",
       "  mean: -0.06470, std: 0.00140, params: {'max_depth': 9, 'min_child_weight': 3},\n",
       "  mean: -0.06436, std: 0.00164, params: {'max_depth': 9, 'min_child_weight': 5}],\n",
       " {'max_depth': 5, 'min_child_weight': 1},\n",
       " -0.06404408813224548)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb2_1 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=93,  #第一轮参数调整得到的n_estimators最优值\n",
    "        max_depth=5,\n",
    "        min_child_weight=1,\n",
    "        gamma=0,\n",
    "        subsample=0.3,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel = 0.7,\n",
    "        objective= 'binary:logistic',\n",
    "        seed=3)\n",
    "\n",
    "\n",
    "gsearch2_1 = GridSearchCV(xgb2_1, param_grid = param_test2_1, scoring='neg_log_loss',n_jobs=-1, cv=kfold)\n",
    "gsearch2_1.fit(X_train , y_train)\n",
    "\n",
    "gsearch2_1.grid_scores_, gsearch2_1.best_params_,     gsearch2_1.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ice2018/anaconda2/lib/python2.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/ice2018/anaconda2/lib/python2.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/ice2018/anaconda2/lib/python2.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/ice2018/anaconda2/lib/python2.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/ice2018/anaconda2/lib/python2.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/ice2018/anaconda2/lib/python2.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/ice2018/anaconda2/lib/python2.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([ 7.1686358 ,  7.25760875,  6.93113403, 10.10597806,  9.89298801,\n",
       "        10.14477   , 13.43325458, 13.18802657, 12.87273664, 14.38455253,\n",
       "        12.29353538, 10.65215726]),\n",
       " 'mean_score_time': array([0.05785193, 0.05686126, 0.06085877, 0.08285193, 0.09058619,\n",
       "        0.07853079, 0.10384459, 0.10292459, 0.11276398, 0.10949783,\n",
       "        0.09166045, 0.08555479]),\n",
       " 'mean_test_score': array([-0.06454811, -0.06455878, -0.0645596 , -0.06404409, -0.06410985,\n",
       "        -0.06413091, -0.06418493, -0.06439329, -0.06414257, -0.0649847 ,\n",
       "        -0.06470092, -0.06436185]),\n",
       " 'mean_train_score': array([-0.06266039, -0.0628472 , -0.06297947, -0.05891998, -0.05972668,\n",
       "        -0.06028622, -0.05388461, -0.05652359, -0.05785141, -0.04926304,\n",
       "        -0.05408144, -0.0564266 ]),\n",
       " 'param_max_depth': masked_array(data=[3, 3, 3, 5, 5, 5, 7, 7, 7, 9, 9, 9],\n",
       "              mask=[False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'param_min_child_weight': masked_array(data=[1, 3, 5, 1, 3, 5, 1, 3, 5, 1, 3, 5],\n",
       "              mask=[False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'params': [{'max_depth': 3, 'min_child_weight': 1},\n",
       "  {'max_depth': 3, 'min_child_weight': 3},\n",
       "  {'max_depth': 3, 'min_child_weight': 5},\n",
       "  {'max_depth': 5, 'min_child_weight': 1},\n",
       "  {'max_depth': 5, 'min_child_weight': 3},\n",
       "  {'max_depth': 5, 'min_child_weight': 5},\n",
       "  {'max_depth': 7, 'min_child_weight': 1},\n",
       "  {'max_depth': 7, 'min_child_weight': 3},\n",
       "  {'max_depth': 7, 'min_child_weight': 5},\n",
       "  {'max_depth': 9, 'min_child_weight': 1},\n",
       "  {'max_depth': 9, 'min_child_weight': 3},\n",
       "  {'max_depth': 9, 'min_child_weight': 5}],\n",
       " 'rank_test_score': array([ 8,  9, 10,  1,  2,  3,  5,  7,  4, 12, 11,  6], dtype=int32),\n",
       " 'split0_test_score': array([-0.06427561, -0.06414767, -0.06416419, -0.06312316, -0.06343918,\n",
       "        -0.06346614, -0.06337079, -0.06297195, -0.06314877, -0.06394447,\n",
       "        -0.06349248, -0.0620595 ]),\n",
       " 'split0_train_score': array([-0.06298147, -0.06315837, -0.06329274, -0.05903275, -0.06000026,\n",
       "        -0.06045848, -0.05413655, -0.05690936, -0.05793578, -0.0498655 ,\n",
       "        -0.05428791, -0.05684963]),\n",
       " 'split1_test_score': array([-0.0660178 , -0.06615879, -0.06626318, -0.06584462, -0.06593052,\n",
       "        -0.06581057, -0.0657473 , -0.06612621, -0.06606299, -0.06623405,\n",
       "        -0.06611001, -0.06634608]),\n",
       " 'split1_train_score': array([-0.06242647, -0.0626604 , -0.06274204, -0.0586302 , -0.05955121,\n",
       "        -0.05995505, -0.05347085, -0.05618264, -0.0572224 , -0.04900943,\n",
       "        -0.05346089, -0.05581874]),\n",
       " 'split2_test_score': array([-0.06538293, -0.0654729 , -0.06560282, -0.06511059, -0.06483351,\n",
       "        -0.06493731, -0.0650464 , -0.06598894, -0.0654762 , -0.06650313,\n",
       "        -0.06666667, -0.06607326]),\n",
       " 'split2_train_score': array([-0.06245514, -0.06261724, -0.06280516, -0.05875789, -0.05924008,\n",
       "        -0.05979642, -0.0535169 , -0.05622342, -0.05770326, -0.04910207,\n",
       "        -0.05406248, -0.05604768]),\n",
       " 'split3_test_score': array([-0.06353835, -0.06355954, -0.06336441, -0.06309087, -0.06322344,\n",
       "        -0.06335626, -0.06407081, -0.06383694, -0.06339152, -0.06452371,\n",
       "        -0.06393473, -0.06405214]),\n",
       " 'split3_train_score': array([-0.06260944, -0.06280642, -0.06295821, -0.05884559, -0.05989223,\n",
       "        -0.06060377, -0.0541564 , -0.05644564, -0.05820727, -0.04885659,\n",
       "        -0.05405802, -0.05668813]),\n",
       " 'split4_test_score': array([-0.06352569, -0.06345481, -0.06340317, -0.06305103, -0.06312244,\n",
       "        -0.06308411, -0.06268923, -0.06304227, -0.06263319, -0.06371804,\n",
       "        -0.06330058, -0.06327818]),\n",
       " 'split4_train_score': array([-0.06282942, -0.06299359, -0.06309922, -0.05933349, -0.05994963,\n",
       "        -0.06061736, -0.05414234, -0.05685687, -0.05818835, -0.04948159,\n",
       "        -0.05453792, -0.05672882]),\n",
       " 'std_fit_time': array([0.30576168, 0.46506512, 0.27185334, 0.32348614, 0.27608704,\n",
       "        0.2372556 , 0.65990823, 0.26589175, 0.59096687, 1.2829919 ,\n",
       "        0.36554509, 1.49937893]),\n",
       " 'std_score_time': array([0.00297336, 0.00432388, 0.00951673, 0.00518635, 0.02573799,\n",
       "        0.00394273, 0.00091765, 0.00591826, 0.01947122, 0.01297241,\n",
       "        0.00518058, 0.00224167]),\n",
       " 'std_test_score': array([0.00099961, 0.00107535, 0.00117581, 0.00119351, 0.00109992,\n",
       "        0.00105919, 0.00110419, 0.00139314, 0.00136357, 0.00116324,\n",
       "        0.00140414, 0.00163937]),\n",
       " 'std_train_score': array([0.00021499, 0.00020381, 0.00019984, 0.00024474, 0.00028966,\n",
       "        0.00034343, 0.00031943, 0.00030735, 0.00036466, 0.000365  ,\n",
       "        0.00035693, 0.00041274])}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch2_1.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best: -0.064044 using {'max_depth': 5, 'min_child_weight': 1}\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAELCAYAAADkyZC4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3XlclOX+//HXNcO+L4IKmOBWpiJq7uKSokKmqVjabqWnOtWpzrHl1DmdzrFOy8ls/33TbLEsE7WswN1cyo5bamrHRDEFVJBNAVnn+v0x4wgIOCLDsHyejwcPZuZe5jNY93vu+74WpbVGCCGEqG8GRxcghBCieZKAEUIIYRcSMEIIIexCAkYIIYRdSMAIIYSwCwkYIYQQdiEBI4QQwi4kYIQQQtiFBIwQQgi7cHJ0AY7UqlUrHR4e7ugyhBCiSdm5c+dprXXQpdZr0QETHh7Ojh07HF2GEEI0KUqp321ZTy6RCSGEsAu7BoxSaqxS6qBSKlkp9VQ1y12VUosty/+rlAqvsCxSKbVVKbVfKfWLUsrN8vr3ln3utvwEX2pfQgghGp7dAkYpZQTeAWKBa4FpSqlrq6x2L5Cjte4EvA68bNnWCfgUuF9r3Q0YDpRW2O42rXWU5Sejtn0JIYRwDHveg+kHJGutjwAopb4AJgAHKqwzAfiH5XEC8LZSSgGjgb1a6z0AWussG96v2n1pmY9AiHpRWlpKamoqRUVFji5FNBA3NzfCwsJwdnau0/b2DJhQ4HiF56lA/5rW0VqXKaXygECgC6CVUquAIOALrfUrFbb7UClVDiwFZltCpKZ9na73TyZEC5Samoq3tzfh4eGYvweK5kxrTVZWFqmpqURERNRpH/a8B1Pdf4FVzyZqWscJGALcZvk9USk10rL8Nq11DyDa8nPHZbwfSqmZSqkdSqkdmZmZl/4UQggAioqKCAwMlHBpIZRSBAYGXtEZqz0DJhVoV+F5GJBe0zqW+y6+QLbl9Y1a69Na60IgEegNoLVOs/w+CyzCfCmutn1VorV+X2t9ndb6uqCgSzbjFkJUIOHSslzpv7c9A2Y70FkpFaGUcgGmAiuqrLMCuMvyOB5Yb7nctQqIVEp5WMJiGHBAKeWklGoFoJRyBsYB+y6xr3pXuGsXZxITMZWU2GP3QgjRLNjtHozlPshDmMPCCCzQWu9XSv0T2KG1XgF8ACxUSiVjPtuYatk2Ryk1B3NIaSBRa/2dUsoTWGUJFyOwFphnectq92UPuUsSyFu+HKOfH74TJuA3JR7XTp3s9XZCCNEk2bUfjNY6UWvdRWvdUWv9guW1v1vCBa11kdZ6ita6k9a63/kWZ5Zln2qtu2mtu2utn7C8VqC17qO1jrQs+5PWuvxS+6pvbV+YTbv58/EYMIDsRYs4Mu5Gjk67ldylyzAVFtrrbYVo8XJzc3n33XfrtO3cuXMpbKT/fw4fPrzOo4p89dVXHDhwoXGuLfsqKiqiX79+9OzZk27duvHcc8/V6b0vRXry14EyGPAaMpiwua/TeeP3BD/xBOV5eZx45hkORQ/lxHP/4Nwv+5AW0kLUr+YaMFeiasDYwtXVlfXr17Nnzx52797NypUr+emnn+q9thY9Fll9cAoIIPCe6QRMv5tzP/9M7pdLyPv6a3IXL8b1mmvwmxKP77hxGH19HV2qEPXm+W/2cyD9TL3u89oQH567sVut6zz11FMcPnyYqKgoYmJiCA4O5ssvv6S4uJiJEyfy/PPPU1BQwM0330xqairl5eX87W9/49SpU6SnpzNixAhatWrFhg0bqt2/l5cXf/zjH1m7di3+/v68+OKLPPHEExw7doy5c+cyfvx4jh49yh133EFBQQEAb7/9NoMGDWL58uW88847rFmzhpMnTzJs2DA2bdpEmzZtLnqfc+fOMX36dA4cOEDXrl05d+6cddnq1at57rnnKC4upmPHjnz44Yd4eXkRHh7OLbfcYq190aJFZGRksGLFCjZu3Mjs2bNZunQpAEuWLOHBBx8kNzeXDz74gOjo6Ervr5TCy8sLMPdvKi0ttUsDDjmDqSdKKTx69ybkpX/TefMm2vzjOZTBwKl/zebQ0GGkPfEEhdu3y1mNEFfgpZdeomPHjuzevZuYmBgOHTrEtm3b2L17Nzt37mTTpk2sXLmSkJAQ9uzZw759+xg7diyPPPIIISEhbNiwocZwASgoKGD48OHs3LkTb29vnn32WdasWcPy5cv5+9//DkBwcDBr1qxh165dLF68mEceeQSAiRMn0qZNG9555x1mzJjB888/X224ALz33nt4eHiwd+9ennnmGXbu3AnA6dOnmT17NmvXrmXXrl1cd911zJkzx7qdj48P27Zt46GHHuLRRx9l0KBBjB8/nldffZXdu3fTsWNHAMrKyti2bRtz587l+eefByA9PZ24uDjrvsrLy4mKiiI4OJiYmBj696/aTfHKyRmMHRi9vfGfOhX/qVM5t38/eUuXkrfiG86s+AaX8HD84ifje9NNOLVq5ehShaiTS51pNITVq1ezevVqevXqBUB+fj6HDh0iOjqav/zlLzz55JOMGzfuom/vtXFxcWHs2LEA9OjRA1dXV5ydnenRowdHjx4FzN/4H3roIXbv3o3RaOS3336zbv/WW2/RvXt3BgwYwLRp02p8n02bNlmDKTIyksjISAB++uknDhw4wODBgwEoKSlh4MCB1u3O73PatGk89thjNe5/0qRJAPTp08dad0hICImJidZ1jEYju3fvJjc3l4kTJ7Jv3z66d+9u09/JVhIwduberRvu3boRPGsWZ1atIjchgYz/vEbG3DfwHjECvynxeA4ejDIaHV2qEE2K1pqnn36aP/zhDxct27lzJ4mJiTz99NOMHj3aevZxKc7OztZLRQaDAVdXV+vjsrIyAF5//XVat27Nnj17MJlMuLm5WbdPS0vDYDBw6tQpTCYTBkPNF4mquySltSYmJobPP//8ktvUdknrfN1Go9Fad038/PwYPnw4K1eurPeAkUtkDcTg7o7fTTcR/umndEj8joA776Rw506Oz/wDyaNiyHzrbUrT0hxdphCNmre3N2fPngVgzJgxLFiwgPz8fMB8cM/IyCA9PR0PDw9uv/12/vKXv7Br166Ltr0SeXl5tG3bFoPBwMKFCykvLwfMl6WmT5/OokWL6Nq1a6VLW1UNHTqUzz77DIB9+/axd+9eAAYMGMAPP/xAcnIyAIWFhZXOkBYvXmz9ff7Mpi6fKzMzk9zcXMB8P2jt2rVcc801l7UPW8gZjAO4duhA6ydmEfzonzi7fgO5CQmcfvddTr/7Lp5DhuAXH4/3iOEoFxdHlypEoxIYGMjgwYPp3r07sbGx3HrrrdYDrZeXF59++inJycnMmjULg8GAs7Mz7733HgAzZ84kNjaWtm3b1nof5lIefPBBJk+ezJIlSxgxYgSenp4AvPjii0RHRxMdHU1UVBR9+/blhhtuoGvXrhft44EHHmD69OlERkYSFRVFv37mAUmCgoL46KOPmDZtGsXFxQDMnj2bLl26AFBcXEz//v0xmUzWs5ypU6cyY8YM3nzzTRISEmqsOz09nfvuu4/ExEROnDjBXXfdRXl5OSaTiZtvvplx48bV+W9SE9WSbzpfd911urHMaFmalkbusuXkLltG2YkTGAMD8b1pAn6T43HtULeB5oSoT7/++mu1B0vRMM7PwNuqge/dVvfvrpTaqbW+7lLbyiWyRsI5NJSghx+i09o1tHv///Do3Zvsjz/hSFwcR2+/ndyvvsJUoSmjEEI0dnKJrJFRRiNeQ4fiNXQoZZmZ5j41SxI48dTTnHrhRXxvHIdffDxu11adu00IYav+/ftbL0Gdt3DhQnr06FGv77Nq1SqefPLJSq9FRESwfPnyy97X+dZgTYlcImskl8hqo7WmcPt2chMSOLtqNbq4GLdu3fCbEo/PuHEYLR2mhLAnuUTWMsklsmZOKYVnv36EvvIKnTdtpPWzz6LLyjj5j+c5FD2U9Kf/SuGuXdKJUwjRqMglsibG6OtLwO234X/brRTt20/ukiWc+fZb8pYvx6VjR/zi4/GdMB6ngABHlyqEaOHkDKaJUkrh3qM7bf/5PJ03b6LtCy9g9PYm4+WXOTRsOKmPPkb+Dz+gTSZHlyqEaKHkDKYZMHh64jd5En6TJ1F86BC5CUvJ+/przq5ciXNoKL6TJ+E3aRLONYyLJIQQ9iBnMM2Ma+fOtH76KTpt2kjonNdwaX8Vp998i+TrR3L8D/dzdu1adGmpo8sUok6a63D9DT0fDJj71fTo0YOoqCiuu+6S9+vrRAKmmTK4uOATF8dVCxbQcc1qAmfOoOjXX0l96GEOXX89Ga/NoeT33x1dphCXpbkGzJWoy3ww523YsIHdu3fXOdwuRS6RtQAu7doR/OijBD30EPmbNpObkEDWggVkzZuHR//+5qFpRsdgsAyQJ8QlJT0FJ3+p33226QGxL9W6iswHUz/zwTQUOYNpQZSTE97Xj6Ddu+/Qaf16gh59lNL0dNJnzeLQ0GGcfOFFig7+dukdCeEgMh9M/c0Ho5Ri9OjR9OnTh/fff/8K/lVqJmcwLZRz62Ba3f8HAmfOoHDbNnKXJJD7xRfkLFyIW2QkfvGT8Ym7AaOXp6NLFY3RJc40GoLMB3Nl88H88MMPhISEkJGRQUxMDNdccw1Dhw616e9kKwmYFk4ZDHgOGIDngAGU5eRw5ptvyF2yhJN/f45TL72MT1ws/vHxuPXsaZcpVYWoK5kP5srmgwkJCQHMZ2QTJ05k27Zt9R4wcolMWDn5+xNw551ErFhB+Bef4xMXy5nEJI5OnUbK+Alkf/IJZTk5ji5TtGAyH0z9zAdTUFBg3aagoIDVq1fX+2RjIGcwohpKKdyjonCPiqL1U09zJvE7chOWcurFf5Px6n/wjonB7+YpePTrh6rlG5oQ9U3mg6mf+WBOnTrFxIkTAXMw3nrrrdZLg/VJBrtsAoNdNhZFBw+SuySBvBUrMJ05g/NVV+E3eTK+E2/COTjY0eUJO5PBLh1L5oMRzZrb1VfT5tln6LxpIyGvvopzmzZkvv46ySOu5/iDf+Tshg3oS8z/LYRoOeQSmbhsBjc3fG8ch++N4yg5epTcpcvIXb6c/PXrcQoOxnfSRPwmT8alXTtHlypEtWQ+mIYhl8jkElm90KWl5G/cSO6SBPI3bwaTCc9BA/GLj8dr1CgMLi6OLlFcIblE1jJdySUyOYMR9UI5O+M9ahTeo0ZRevIkucuWkZewlLTH/4zRzw/fCRPwmxKPa6dOji5VCNFA5B6MqHfObdoQ9OCDdFy7hnYfzMdjwACyFy3iyLgbOTrtVnKXLsPUDMeEEkJUJmcwwm6UwYDX4MF4DR5MWXY2eV99TW5CAieeeYZTL76Iz7hx+MXH49a9m3TiFKIZkjMY0SCcAgIIvGc6Hb77lvaLPsN79Gjyvv6ao1OmkDJxEtmffkZ5Xp6jyxSNXHMdTbmhh+s/ePAgUVFR1h8fHx/mzp1bp/evjQSMaFBKKTx69ybk3y/SefMm2vzjOZTRyKnZszk0dBhpTzxBwbZttOTGJ6JmzTVgrkRdhuu/+uqr2b17t3WQUA8PD2vHy/okASMcxujtjf/UqUQsTSBi2VL8Jk8if8P3HLvzLo6MjSVr/nzKTp92dJmiEak4XP+sWbN49dVX6du3L5GRkTz33HOAeeiTG264gZ49e9K9e3cWL17Mm2++aR2uf8SIETXu38vLiyeffJI+ffowatQotm3bxvDhw+nQoQMrVqwAzM2Fo6Oj6d27N7179+bHH38EYPny5YwaNQqtNSdOnKBLly6cPHmy2vc5d+4cU6dOJTIykltuueWi4foHDhxI7969mTJlinUonPDwcJ588kn69etHv379SE5O5scff2TFihXMmjWLqKgoDh8+DJiH6+/Xrx9dunRh8+bNtf5N161bR8eOHWnfvr2N/wq2s+s9GKXUWOANwAjM11q/VGW5K/AJ0AfIAm7RWh+1LIsE/g/wAUxAX611UYVtVwAdtNbdLc+jgP8HuAFlwINa6232/Hyi/rhdey1t/v53gmfN4syqVeQmJJDxn9fImPsG3iOG4zdlCp6DB6OMRkeXKoCXt73M/7L/V6/7vCbgGp7s92St67z00kvs27eP3bt3s3r1ahISEthmOeMdP348mzZtIjMzk5CQEL777jvAPHaYr68vc+bMYcOGDbX2hD8/XP/LL7/MxIkTrcP1HzhwgLvuuovx48dbh+t3c3Pj0KFDTJs2jR07djBx4kSWLl3KO++8w8qVK20ern/v3r307t0bqDxcv6enJy+//DJz5syxDtZ5frj+Tz75hEcffZRvv/2W8ePHM27cOOLj4637Pz9cf2JiIs8//zxr166tNFRMRV988UWtIz9fCbsFjFLKCLwDxACpwHal1AqtdcVzuXuBHK11J6XUVOBl4BallBPwKXCH1nqPUioQKK2w70lAfpW3fAV4XmudpJSKszwfbqePJ+zE4O6O30034XfTTRQfOUJuwlLyvvqKs2vW4tS2LX6TJuE3aSLOoaGOLlU4mAzXf2XD9Z/f/4oVK/j3v/99yb9NXdjzDKYfkKy1PgKglPoCmABUDJgJwD8sjxOAt5W5OdFoYK/Weg+A1jrr/AZKKS/gcWAm8GWFfWnMZzsAvkB6PX8e0cBcO3Sg9ROzCH70T5zd8D25S5Zw+t13Of3uu3gOHozflCl4jxiOkk6cDe5SZxoNQYbrv7Lh+gGSkpLo3bs3rVu3rnGdK2HPezChwPEKz1Mtr1W7jta6DMgDAoEugFZKrVJK7VJKPVFhm38BrwFV79Y9CryqlDoO/Ad4ur4+iHAs5eKCz5jRXDV/Hp3WrqHVgw9SfPgwaX/6E4eGj+DUK69SfOSIo8sUDUCG66+f4frP+/zzz+12eQzsGzDVxWvVpkE1reMEDAFus/yeqJQaabnP0klrXd1APg8Aj2mt2wGPAR9UW5RSM5VSO5RSOzIzM238KKKxcA4NJejhh+i0dg3t3v8/PPr0IfuTTzgSdwNHb7+d3K++wlThhqloXioO179mzRrrcP09evQgPj6es2fP8ssvv9CvXz+ioqJ44YUXePbZZ4ELw/XXdpPfFg8++CAff/wxAwYM4Lfffqt2uP45c+Ywf/58fv3112r38cADD5Cfn09kZCSvvPJKtcP1R0ZGMmDAAP73vwv3us4P1//GG2/w+uuvA+bh+l999VV69eplvclfnapTJhcWFrJmzRrr5TR7sNtYZEqpgcA/tNZjLM+fBtBa/7vCOqss62y13Hc5CQQBtwBjtdZ3W9b7G1CE+b7L34ASzCEUDPyotR6ulMoD/LTW2nKZLU9rff6SWbVkLLLmoSwzk7yvvyZ3SQIlv/+Owdsb3xstnTivvdbR5TUbMhaZY8lw/ZVtBzorpSKUUi7AVGBFlXVWAHdZHscD67U58VYBkUopD0vwDAMOaK3f01qHaK3DMZ/Z/Ka1Hm7ZPt2yHsD1wCE7fS7RyDgFBRF43310WJlE+4Wf4DViOLlLl5EyaTIpkyaT88UXlNfDpREhxOWx201+rXWZUuohzGFhBBZorfcrpf4J7NBar8B8GWuhUioZyMYcQmitc5RSczCHlAYStdbfXeItZwBvWAKpCHMjANGCKKXw6NsXj759KX/mGfK+/ZbcJQmc/MfznHrpZXzGjsXv5im49+olQ9O0cDJcf8OQ4frlElmzprWmaN9+cpcs4cy332IqLMSlQwf84uPxvWkCTgEBji6xyZBLZC1TY71EJoTDKaVw79Gdtv98ns6bN9H2hRcw+vqS8corHBo2nNRHHyN/yw9ok8nRpQrR7MhoyqLFMHh64jd5En6TJ1F86JC5E+fXX3N25UqcQ0LwjZ+M36RJONfQ+1oIcXnkDEa0SK6dO9P66afotGkjoXNewyW8PafffIvk60dy7A9/4OzatejS0kvvSAhRIzmDES2awcUFn7g4fOLiKDl+nNylS8lbtpzUhx7G2KoVfhNvwi8+Hhc7DAQoRHMnZzBCWLi0a0fwo4/Saf06wt57F/eePcla8CGHx4zl9zvvIu+bbzFVaXkkGlZzHa6/oeeDAXjjjTfo3r073bp1s8tcMCABI8RFlJMT3iNG0O6dt+m0fj1Bjz1G6YkTpM+axaGhwzg5+wWKDh50dJktUnMNmCtRl/lg9u3bx7x589i2bRt79uzh22+/5dCh+u86KAEjRC2cWwfT6g8z6bhqJVd99CFeQ4aQu3gxKRNuIuXmW8j58kvK8wscXWaLIfPB1M98ML/++isDBgzAw8MDJycnhg0bVqe+OZci92CEsIEyGPAcMADPAQMoy8nhzDffkLtkCSf//py5E2dcLP7x8bj17NkiOnGefPFFin+t3/lgXLteQ5u//rXWdWQ+mPqZD6Z79+4888wzZGVl4e7uTmJiItddd8luLZdNAkaIy+Tk70/AnXfif8cdFO3dS86SJZxJTCIvYSmunTvjNyUenxtvxMnf39GlNmsyH0zd54Pp2rUrTz75JDExMXh5edGzZ0+cnOo/DiRghKgjpRTuPXvi3rMnrZ96mjNJieQuSeDUi/8m49X/4B0Tg9+UeDz690fVMi9IU3SpM42GIPPBXNl8MPfeey/33nsvAH/9618JCwurcX911bz+qxfCQYxenvhPmULEl4uJ+Por/G65hfwtWzg2/R4OjxnL6f/3f5SeynB0mU2ezAdTf/PBZGSY/3s8duwYy5Yts8u8MHIGI0Q9c7v6ato8+wzBf/kzZ9esJTchgcy5c8l86y28hg3DLz4er6HRKDtckmjuKs4HExsba50PBsw36D/99FOSk5OZNWsWBoMBZ2dn3nvvPeDCfDBt27Zlw4YNda7hwQcfZPLkySxZsoQRI0ZUOx9MVFQUffv25YYbbqh2/LYHHniA6dOnExkZSVRUVLXzwZwfjHP27Nl06dIFuDAfjMlksp7lTJ06lRkzZvDmm2+SkJBQY90V78EATJ48maysLJydnXnnnXfwt8MlXRnsUga7FA2g5OhRcpcuI3f5cspPn8YpOBjfSRPxmzwZl3btHF2eTWSwS8eS+WCEENVyCQ8n+M+P03nDesLefgu3rl3Jen8eh2NG8/v06ZxJTMRUUuLoMoWoV3KOLkQDUs7OeI8ahfeoUZSePEnusmXkJSwl7fE/Y/Tzw3fCePzi43Ht3NnRpTZrMh9Mw5BLZHKJTDiYNpko2LqV3IQEzq5dB6WluPfqRcA90/EeObLRtECTS2Qt05VcIpMzGCEcTBkMeA0ejNfgwZRlZ5P39QpyFi0i7eFHcOnQgcB778X3xnEoFxdHl4rWukV0JBVmV3oC0ji+GgkhAHAKCCBw+t10TEokdM5rKFdXTjzzDMkxo8n66CNMBY4blsbNzY2srKwrPuiIpkFrTVZWVqV+PpdLLpHJJTLRiGmtKdjyA1nz5lG4bRsGX18CbrsV/9tvb/DpnktLS0lNTaWoqKhB31c4jpubG2FhYTg7O1d63dZLZBIwEjCiiTi3Zw9Z8+dzds1alJsbfvHxBE6/G+fQUEeXJloYCRgbSMCIpqj48GGyPlhA3ooVoDW+424g4N57cbN0xhPC3iRgbCABI5qy0hMnyP7oY3KWLEEXFuI1YgSBM+7DwzIyrxD2IgFjAwkY0RyU5+aSvWgROZ8spDw3F/c+fQiccR9ew4ZJiy9hFxIwNpCAEc2JqbCQ3KXLyPpwAWXpJ3Dt0oXAGffhExsr456JeiVDxQjRwhg8PAi443Y6rVpFyMsvgTaRPusJDo8ZS/Znn2GqMGuiEA1BzmDkDEY0U9pkIv/7jWTNm8e5n3/GGBBAwB2343/rrRh9fR1dnmjC5BKZDSRgREtRuHMnWe/PI3/jRgweHvjdcgsBd9+Fc+vWji5NNEESMDaQgBEtTdHBg2TN/4AziYlgMOA7YTyB99yLa4cIR5cmmhAJGBtIwIiWqiQ1lewFH5K7dCm6pATvmBgCZ9yHez2PJiyaJwkYG0jAiJauLCuL7IULyVn0OaYzZ/AYMIDAGffhOWiQNHEWNZKAsYEEjBBm5fkF5H75JdkffURZRgZu115L4MwZeMfEoIxGR5cnGhlppiyEsJnRy5PAe6bTce0a2s7+F6bCQtIefYzDcXHkfPmlzLYp6kQCRghhZXBxwS8+ng7ffUvoG29g9Pbh5N+f4/DIUWTNn095fr6jSxRNiF0DRik1Vil1UCmVrJR6qprlrkqpxZbl/1VKhVdYFqmU2qqU2q+U+kUp5VZl2xVKqX1VXnvY8n77lVKv2OtzCdHcKaMRnzGjCV/yJVd9uADXzp3I+M9rJI+4now5r1N2+rSjSxRNgN3Gj1BKGYF3gBggFdiulFqhtT5QYbV7gRytdSel1FTgZeAWpZQT8Clwh9Z6j1IqECitsO9JQKWvUkqpEcAEIFJrXayUCrbXZxOipVBK4TlwIJ4DB3Ju336y5s8na948sj/6CN/Jkwi85x5c2rVzdJmikbrkGYxSqqNSytXyeLhS6hGllJ8N++4HJGutj2itS4AvMAdARROAjy2PE4CRytx0ZTSwV2u9B0BrnaW1LrfU4AU8Dsyusq8HgJe01sWWbTJsqFEIYSP37t0Im/s6HRK/w3fCBPISlnJ4zFjS/vwXiv73P0eXJxohWy6RLQXKlVKdgA+ACGCRDduFAscrPE+1vFbtOlrrMiAPCAS6AFoptUoptUsp9USFbf4FvAYUVtlXFyDacqlto1Kqrw01CiEuk2tEBG3/9U86rl1LwPS7yf/+e1JumsixGTMp2LZNplQWVrYEjMly8J8IzNVaPwa0tWG76hrRV/0vr6Z1nIAhwG2W3xOVUiOVUlFAJ6318mq2cwL8gQHALOBLVU1DfqXUTKXUDqXUjszMTBs+hhCiOs6tg2k9axadNqwn6LHHKDpwgGN33sXvU6dxdt06tMnk6BKFg9kSMKVKqWnAXcC3lteca1n/vFSg4sXZMCC9pnUs9118gWzL6xu11qe11oVAItAbGAj0UUodBbYAXZRS31fY1zJttg0wAa2qFqW1fl9rfZ3W+rqgoCAbPoYQojZGHx9a/WEmndatpc1zf6csK4vUPz6XCNuuAAAgAElEQVTEkRvHk7v8K7Q0cW6xbAmY6ZgP7C9orVOUUhGYb8Bfynags1IqQinlAkwFVlRZZwXm4AKIB9Zr8/n1KiBSKeVhCZ5hwAGt9Xta6xCtdTjmM5vftNbDLdt/BVwPoJTqArgA0tRFiAZicHPDf9o0Oq5MIuQ//0E5OXHi6adJHjOW7I8/xlRQ4OgSRQO7rJ78Sil/oJ3Weq+N68cBcwEjsEBr/YJS6p/ADq31CkvT44VAL8xnLlO11kcs294OPI35klmi1vqJKvsOB77VWne3PHcBFgBRQAnwF631+trqk578QtiP1pqCzZvJmjefwu3bMfr64n/77fjffhtO/v6OLk9cgXobKsZyCWo85nscu4FMzJevHq+HOh1KAkaIhlH4889kzf+A/HXrUO7u+E2JJ/Duu3EOCXF0aaIO6nOoGF+t9RlgEvCh1roPMOpKCxRCtBwevXrR7p236fDtN/iMGUPOos9JHj2G9Keepjg52dHlCTuxJWCclFJtgZu5cJNfCCEum2unToS89G86rV6F/63TOLNqFUfG3cjxB/9I4c8/O7o8Uc9s6cn/T8w33X/QWm9XSnUADtm3LCFEc+YcEkKbv/6VVg88QM6nn5Hz6af8vn49HtddR+DMGXhGR9f7dAEVbwdUvTOga1rvon1U3EbXuKy21ytuVx91VF1Y0/6r7sPL1QkXJ/sORynD9cs9mEbPZNKUmTTlJk2ZyUS55XF5hdfLq6xjMmFdt8ykq+xDU24yUX6JdUwV1q34PtW/r+kS+6i99ur2eUHNB7KaDkgXL6v8N63x4GXjAa+2/dd24L3oaGN5wa2smFEpP3HToe8JOpfHEd8QlnYZwZbQnpgMxjp/TlGzj6b3ZfjVdRtRqz5v8ocBbwGDMf9bbgH+pLVOrVNljUhjDZiqB6WqB8u6HVArH+RsOaDWvA/bD6g11X45+2xMnI0Ko0HhZDBgUOBkNFieKwxK4WRdXvG5ASeDwqgsy2pZx6AsrxsUFb/AV/0uX3mZqvb1qtvVdkZQ0/4uXlbLdhWeXNzD2bb9G8pKCd21mU7rluN9KpWCwNYcuf4mjvcbgXZxq3abi96vljrq8jkveq86/B1r/3epuqzm7Wqqo/bPWf02I7sGE+bvUfMb1KI+A2YN5qFhFlpeuh24TWsdU6fKGpG6BswX246x8bfMFnVAvXBwNFR5bvsBtfLzygdUo1FV+9y8jaGGfdR+kK68j2rWqXYf5vAwqgrbW/ZhMMgMjw1Fm0zkb9hA1vvzOLdnD8bAQALuuAP/W6dh9PFxdHktXn0GzG6tddSlXmuK6howb647xDd70isdtIwK84GplgOqUXHh4GussPwyDqgX77OmfVSzTo0HaTmgisZJa825HTs4PW8eBZs2Y/D0xG/qLQTceRfOrWXAdEepz4BZC3wEfG55aRowXWs98kqLdLTGeolMCHGxol9/JWv+B5xJSkIZjfjedBOB996DS3i4o0trceozYK4C3sY8XIwGfgQe0Vofq49CHUkCRoimp+T4cbIWLCBv6TJ0aSneY8YQeN99uHfv5ujSWox6C5gadv6o1npunSprRCRghGi6yk6fJvuTheQsWoQpPx/PQQMJnDEDjwED6r2Js6jM3gFzTGt9VZ0qa0QkYIRo+srPniV38WKyPv6Y8szTuHXvTuCMGXiPGokyGh1dXrNUn0PFVLv/Om4nhBD1yujtTeB999Fp7Vra/PN5ys+eIe1Pf+LIDePITUjAJNMFOExdA6ZxtaUVQrR4BldX/G++mY6JiYTOfR2Dhwcnnv0bh0fFkPXBAsrzZbqAhlbjJTKl1FmqDxIFuGutbRlmplGTS2RCNF9aawq3buX0vHkUbv0Jg48P/rdOI+COO3AKDHR0eU2aXe/BNBcSMEK0DOd++YWsefM5u2YNysUFv8mTCbhnOi5hYY4urUmSgLGBBIwQLUvxkRSyFnxA3tcrwGTCJy6OwPvuxe3qqx1dWpMiAWMDCRghWqbSU6fI/uhjchcvxlRYiOewobSaMQP3Pn2kibMNJGBsIAEjRMtWnpdHzuefk/3xJ5Tn5ODeqxeBM2bgNXwYymDfoeybMns3UxZCiCbP6OtLq/vvp9P6dbT+27OUnTpF6oMPkjJhAnlff40uLXV0iU3aJQNGKXVWKXWmys9xpdRyy+RjQgjRpBnc3Qm47TY6rlpJyKuvAIr0J58iecwYsj9ZiKmw0NElNkm2jEX2PJCOech+BUwF2gAHgQe01sPtXKPdyCUyIUR1tNbkb9xI1rz5nNu5E6OfH/533I7/rbfi5O/v6PIcrj4Hu/yv1rp/ldd+0loPUErt0Vr3vMJaHUYCRghxKYW7dpE1bz75GzagPDzwnzKFgOl349ymjaNLc5j6vAdjUkrdrJQyWH5urrCs5bYQEEK0CB69e9PuvXeJWPE1PjGjyP70U5JjRpP+12coPnzY0eU1aracwXQA3sA8XD/AVuAxIA3oo7XeYtcK7UjOYIQQl6s0LY2sDz8iNyEBXVSE16iR5ibOPZvsxZzLJs2UbSABI4Soq7LsbHI+/Yzszz7DlJeHR79+BM6YgeeQwc2+L029XSJTSoVZWoxlKKVOKaWWKqVkfAUhRIvmFBBA0CMP03n9OoKfepKS33/n+IwZpEyazJnERHRZmaNLdDhb7sF8CKwAQoBQ4BvLa0II0eIZPD0JvPtuOq1ZTdsXXkAXFZH2+J85HHcDOV98gam42NElOowt92B2a62jLvVaUySXyIQQ9U2bTJxdt46sefMp2rsXY6tWBNx5J/7TpmL09nZ0efWiPluRnVZK3a6UMlp+bgeyrrxEIYRofpTBgE9MDOGLv+Cqjz7C7ZpryJwzh+QR15Px2muUZWY6usQGY8sZzFXA25hbkWngR+ARrfUx+5dnX3IGI4RoCEUHDpA1fz5nVq5COTnhO3EigfdMx6V9e0eXVid2bUWmlHpUaz23TpU1IhIwQoiGVPL772Qt+JC85cvRZWV4jxlN4H334d6tm6NLuyz2DphjWuur6lRZIyIBI4RwhLLMTLI/WUjO559jys/Hc/BgAmfMwKN/vybRxNneoyk3/r+AEEI0Uk5BQQT/+XE6bVhP0J8fp+jgQY7dfTdHb5nKmTVr0CaTo0usF3UNGJtOe5RSY5VSB5VSyUqpp6pZ7qqUWmxZ/l+lVHiFZZFKqa1Kqf1KqV+UUm5Vtl2hlNpXzT7/opTSSqlWl/+xhBCi4Ri9vWk1Ywad1q2lzT/+QXlODmkPP8KRG8aRu3QZuqTE0SVekRoDpoZh+s8opc5i7hNTK6WUEXgHiAWuBaYppa6tstq9QI7WuhPwOvCyZVsn4FPgfq11N2A4YJ2YQSk1Cciv5j3bATFAk2+AIIRoOQyurvhPvYWOSYmEznkN5e7GiWeeITlmNFkffkR5foGjS6yTGgNGa+2ttfap5sdba+1kw777Acla6yNa6xLgC2BClXUmAB9bHicAI5X5AuRoYK/Weo+lliytdTmAUsoLeByYXc17vg48gQzCKYRogpSTEz5xcUQsXUq7+fNxCQ8n4+WXSR45ksw336QsO9vRJV4We85oGQocr/A81fJatetorcuAPCAQ6AJopdQqpdQupdQTFbb5F/AaUGkGIKXUeCDtfCgJIURTpZTCa8hg2n/8EeGLv8CzX19Ov/seydeP5OTsFyhNS3N0iTax5UykrqprCFD1zKKmdZyAIUBfzEGyTim1E3MHz05a68eq3K/xAJ7BfOZTe1FKzQRmAlx1VZNvCCeaIZM2cbbkLL6uvo4uRTQC7j17EvbWWxQfPkzWBwvIWbyYnM8/x+eGOALvuw+3Ll0cXWKN7BkwqUC7Cs/DMM+MWd06qZb7Lr5AtuX1jVrr0wBKqUSgN+b7Ln2UUkcttQcrpb4HHgYigD2WJn5hwC6lVD+t9cmKb6i1fh94H8zNlOvrwwpxJfKK89iavpXNaZvZkraF7KJsegb1JDYiljHhY2jlLm1WWjrXjh0JefEFgh5+iOyPPiZnyRLOrPgGr+HDCZw5A4/evR1d4kXsNly/JTB+A0ZinjtmO3Cr1np/hXX+CPTQWt+vlJoKTNJa36yU8gfWYT6LKQFWAq9rrb+rsG048K3Wuns1730UuO58QNVE+sEIR9FaczDnIJtTN7M5bTN7Mvdg0iZ8XX0ZFDKI9j7tWX9sPb/l/IZBGejbpi+x4bGMaj9KzmwEAOW5uWQvWkTOJwspz83FvU8fAmfch9ewYXbvS9Mo5oNRSsUBcwEjsEBr/YJS6p/ADq31CkvT44VAL8xnLlO11kcs294OPI35klmi1vqJKvsORwJGNCFnS86yNX0rW9K2sCVtC5nnzGNSXRt4LUNChxAdGk2PVj0wGozWbQ7nHiYpJYmklCSOnT2Gk8GJwSGDiY2IZUS7EXg4ezjq44hGwlRYSO7SZWR9uICy9BO4dulC4Iz78ImNRTnZ5yJVowiYxk4CRtiT1ppDuYfYkraFzamb2Z2xmzJdhrezN4NCBxEdGs3g0ME2Xf7SWnMg+wBJR5JIOppERmEGbkY3hrUbRmxELNGh0bgYXRrgU4nGSpeWciYxkaz58yk+lIxzSAgB99yD3+RJGNzd6/W9JGBsIAEj6ltBaQE/nfjJGiqnCk8BcLX/1USHRRMdGk1kUCROhrp/szRpEz9n/ExSShKrj64mpzgHb2dvrr/qeuIi4ujXtt8V7V80bdpkIn/jRrLen8e5n3/G6O9PwJ134H/rrRh96+fyqgSMDSRgxJXSWpOSl8LmtM1sTt3MzoydlJnK8HT2ZGDbgUSHRTM4ZDCtPVvb5f1LTaVsO7GNxJRE1h9bT35pPgFuAcS0jyEuIo6o4CgMyp69EURjVrhzJ1nvzyN/40YMHh743XILAXffhXPrK/vvUQLGBhIwoi4KSwvZfnK7NVTSC8yNIzv5dSI6NJrosGiigqJwNjo3aF3F5cVsSd1CYkoiG1M3UlxeTBvPNowNH0tsRCxdA7o2iYEURf0rOniQrPkfcCYxEQwGfMffSKv778elXbtLb1wNCRgbSMAIW/1+5ndri68dJ3dQYirB3cmdAW0HWG/Qt/Vq6+gyrQpKC9hwfANJKUn8mPYjZbqMcJ9wxkaYw6aDbwdHlygcoCQ1lewFH5K7dClXLfgAjz596rQfCRgbSMCImhSVFbH95HbzvZS0zRw/ax6UIsI3gujQaIaEDqFP6z5N4sZ6XnEea35fQ1JKEttPbkejudr/amIjYomNiCXE65JDC4pmpjw3F6OfX523l4CxgQSMqOj42eNsTjV3dNx2chvF5cW4Gd3o17afNVTCvMMcXeYVySzMZNXRVSQdTWJv5l4A6dApLpsEjA0kYFq2kvISdpzaYQ2Vo2eOAnCV91XWFl99WvfBzcmt9h01UalnU1l5dCVJKUmVOnTGRcQx8qqR0qFT1EgCxgYSMC1Pen66tQnxf0/+l3Nl53AxuNC3TV+iw8xnKe19muY86VficO5hElMSSUpJ4vjZ4zgZnBgSMoTYiFiGtxsuHTpFJRIwNpCAaf5Ky0v5OeNna4uvw3mHAQj1CmVI6BCGhg2lb5u+uDvVb0e0pkprzYGsAySmJLLy6EoyCjNwd3JnWJi5Q+eQ0CFN4r6TsC8JGBtIwDRPpwpOWW/Ob03fSmFZIU4GJ65rfZ25xVdYNBE+EdJk9xJM2sSuU7vMHTp/X01ucS7ezt6MbD+S2IhY+rWRDp0tlQSMDSRgmodSUyl7MvZYQ+W3nN8AaOPZxnpzfkDbAXKZ5wqUmkr574n/kpSSVKlD5+j2o4nrEEfPoJ7SobMFkYCxgQRM05VZmGkdNHJr+lbOlp7FSTnRq3Uva6h08uskZyl2UFxezObUzSSlJFk7dLb1bMvY8LGMjRgrHTpbAAkYG0jANB3lpnJ+Of0Lm1I3sSVtC79m/wpAkHuQtcXXgLYD8HLxcnClLUtBaQHrj61n5dGVlTp0xkbEMjZirHTobKYkYGwgAdO4ZZ3L4sf0H9mcupkf0n/gTMkZjMpIz6Ce1lDp4t9Fvi03ErlFuaw9trZSh85rAq4xh034WOnQ2YxIwNhAAqZxMWkT+0/vt7b42p+1H40m0C2QwaGDiQ6LZmDbgdI/ownIKMxg9dHVJKUksfe0uUNnVFAUYyPGSofOZkACxgYSMI6XW5TLD+k/sCVtCz+k/UBOcQ4KRWRQpLXFV9eArnIDuQk7fvY4q46uIjElkUM5hzAoA/3a9CM2IlY6dDZREjA2kIBpeCZt4tfsX9mSam7x9cvpXzBpE/6u/uazlNBoBoUMws+t7uMkicYrOSeZpKNJlTt0hg4hNlw6dDYlEjA2kIBpGHnFeWw9sZUtqeZWX1lFWQB0D+xu7T3fLbBbpamCRfNWU4fO4WHDGRsxVjp0NnISMDaQgLEPrTW/5fxmvZeyJ3MP5bocHxcfBoeY76UMChlEoHugo0sVjUC1HTpdvBl11SjGRoyVDp2NkASMDSRg6k9+ST4/nfiJzWmb2ZK6hYxzGQB0DehqHZKle6vucqAQtarYoXPdsXUUlBYQ4BbAmPAxxEbESofORkICxgYSMHWntSY5N9nae/7nUz9TpsvwdvZmYMhAhoQOYUjoEII8ghxdqmiiisqK2JJmnqFzU+qmSh06YyNiuSbgGmmi7iASMDaQgLk8haWF/HTiJ2uonCw4CUAX/y7W3vM9g3vibGjYqYJF85dfkm+doXNr+tZKHTpjI2KJ8I1wdIktigSMDSRgaqe1JuVMinW+lJ2ndlJqKsXDyYOBIQOJDo1mcOhg2ni2cXSpogXJLcplzTHzDJ07Tu5Ao+ka0NU8HXR4bKOaurq5koCxgQTMxc6VnWP7ye3WIVnS8tMA6Ojb0dp7vldwL5yNcpYiHC+jMMM8Q2dKEr+c/gUwd+iMjYhldPho6dBpJxIwNpCAMTt25pi1xdf2k9spMZXg7uRO/zb9rc2IZZgP0dgdP3uclSkrSTqaVKlDZ1xEHCPbj8THxcfRJTYbEjA2aKkBU1xezI6TO6yhcuzsMQDCfcKtvef7tO6Dq9HVwZUKUTfJOcnWGTpT81NxNjgzOHQwcRFxDAsbJh06r5AEjA1aUsCknk01NyFO28K2E9soKi/C1ehKvzb9zKESGk07n3aOLlOIeqW1Zn/WfhJTElmVsoqMcxc6dMZGxDI4dLB06KwDCRgbNOeAKSkvYeepndYWXyl5KQCEeYUxNGwoQ0KH0LdNX9yc3BxcqRANw6RN7Dy1k6SUJNb8vqZSh87YiFj6tukr/bRsJAFjg+YWMCfyT1jPUn468RPnys7hbHCmb5u+1mbE7X3aS98B0eKVmkr5Kf0n8wydx9dX6tAZFxFHZFCkdOishQSMDZp6wJSaStmdsZvNqZvZnLaZ5NxkAEI8Q6wtvvq26SvXm4WoRVFZEZvTzDN0VurQGTGWuIg4rva/Wr6UVSEBY4OmGDAZhRnmy16pm9l6YisFpQU4GZzoE9zHGioRvhHyP4QQdVBdh84I3whiw80dOsN9wx1dYqMgAWODphAwZaYy9mTusYbKwZyDALT2aG1t8TWg7QA8nT0dXKkQzUtOUQ5rfl/DyqMrK3XoPD9DZ0vu0CkBY4PGGjCnz51mS5p5aPsf03/kbMlZjMpIr+Be1n4pnf06y1mKEA3kVMEpVh1dxcqjK60dOnsF9yI2IpaY9jEtrkNnowgYpdRY4A3ACMzXWr9UZbkr8AnQB8gCbtFaH7UsiwT+D/ABTEBfrXVRhW1XAB201t0tz18FbgRKgMPAdK11bm31NZaAKTeV88vpX6w36A9kHQCglXsrokOjrWcp3i7eDq5UCHH8zHFWHl1JYkoiybnJGJSB/m36m2fobCEdOh0eMEopI/AbEAOkAtuBaVrrAxXWeRCI1Frfr5SaCkzUWt+ilHICdgF3aK33KKUCgVytdbllu0lAvGXb8wEzGlivtS5TSr0MoLV+srYaHRkw2UXZ/JD2A5vTNvNj+o/kFedhUAZ6BvW0tviS0WKFaNwO5RwiKSWpUofOIaFDiIuIY2jY0GbbwMbWgLFno+9+QLLW+oiloC+ACcCBCutMAP5heZwAvK3MR9TRwF6t9R4ArXXW+Q2UUl7A48BM4Mvzr2utV1fY70+YA6jRMGkTB7IOWFt87Tu9D40mwC2AYWHDiA6NZmDIQJmfXIgmpLN/Zzr7d+bhXg+z7/Q+ko4msSplFRuObzB36Gw3nNjwltuh054BEwocr/A8Fehf0zqWM488IBDoAmil1CogCPhCa/2KZZt/Aa8BhbW89z3A4iv+BFcorziPH9N/ZHPqZn5I/4HsomwUih5BPXgg6gGGhg6la2BXaW8vRBOnlPn/6x5BPfhznz+zK2OXtUNnUkoS3i7exLSPYWy4eYbOljI9uD0DprprO1Wvx9W0jhMwBOiLOUjWKaV2Yr5P00lr/ZhSKrzaN1XqGaAM+KyG5TMxn/1w1VVXXfJDXA6TNvG/7P9ZW3ztPb0Xkzbh5+rHoJBBRIdFMzhkMP5u/vX6vkKIxsNoMNK3TV/6tunL0/2ftnboXJmykmWHlhHoFlhphs7mfBncngGTClQc3CoMSK9hnVTLfRdfINvy+kat9WkApVQi0BvIB/oopY5aag9WSn2vtR5uWe8uYBwwUtdwc0lr/T7wPpjvwVzphzxTcoat6Vutrb5OnzsNQLfAbsyMnMmQ0CF0D+zeYr6xCCEucDY4m/unhUVX6tCZ8FsCi/63iBDPEPM8NhGxzbJDpz1v8jthvsk/EkjDfJP/Vq31/grr/BHoUeEm/ySt9c1KKX9gHeazmBJgJfC61vq7CtuGA99WuMk/FpgDDNNaZ9pSY11v8h/OPcyG4xvYkraF3Rm7KdfleLt4MzhkMNFh0QwKGdTimi0KIWx3vkNnYkoiW9O3Uq7Lm1SHToe3IrMUEQfMxdxMeYHW+gWl1D+BHVrrFUopN2Ah0AvzmcvUCo0CbgeexnzJLFFr/USVfYdTOWCSAVfMl9EAftJa319bfXUNmH9t/Rdf/vYl1wRcY23xFRkUKQPlCSEu2/kOnUkpSew8tbNSh87YiNhGOWNsowiYxq6uAXMi/wRGg5Fgj2A7VCWEaKnOd+hMSkliX9Y+AHoH92ZsxFhGtx9NoHuggys0k4CxQWPpaCmEEFUdP3OcpKPmPjbJuckYlZH+bfszNnyswzt0SsDYQAJGCNEUVNehMzo0mtiIWIa1G4a7k3uD1iMBYwMJGCFEU6K1Zt/pfeYZOo+uIvNcJu5O7oxoN8I8Q2fIYJyNznavQwLGBhIwQoimqtxUzq6MXSSmJLLm9zXkFedZO3TGRsTSt3Vfu3WPkICxgQSMEKI5KDWVsjV9q3mGzmPrKSwrtGuHTgkYG0jACCGam6KyIjalbrLO0FliKrF26IyLiKOLf5crDhsJGBtIwAghmrP8knzWH19vnaHT2qEzIpaJnSbWuY9NYxhNWQghhAN5uXgxvuN4xnccX6lD53u736N3cG+7d+KUMxg5gxFCtDAn808Q5B6E0Vi3cww5gxFCiMZGaygvgbIiKLP8tj4vNv+UF19i2fnHtSy7aD+Vl7UpL4bbEqDTSLt+XAkYIUTzZyq3HFyLKh/gKx14i2o4+Ne2rKjKwb7qsirPy0vq5/MYnMDJDYwu4ORq/jG6Xnjs5AbOfubfTi5VllnW9WtfP7XUQgJGCGEfWkN5aa3fpCt/I69lWa3BcH7d6oLBssxUVj+fyeha+0HbyRXcfC4+4Ne0bo3LLO9RU4g0kek/JGCEaG5MpsrfmCsdeKu7xFLbwd+WYKhlPxfNMVgHymA54J4/wJ4/8FY4KLt6g0erCgdilwvbXHRAr2VZbQd/ows0s/la7E0CRghHKj0H+RlQkAn5py48Lsyqw6Uay29Taf3UZnSp4cBc4Zu1h1fNyyo9d73EsloO/nW8ES0cT/7lhKhv50MjPwMKMqoPkPPLS85Wvw9XH8uBtpqDtrM7uPtXc+mkhgN8bctqOvgbXcBgaNi/m2h2JGCEsEVJoSUsMi+ERrUBkllzaLj7g2cweAVDSJTlcZDld+sLjz2DzAd8IZo4CRjRclUMjfxTVQLkVIXHtoZGrwuh4dW6coBIaIgWSAJGNC/W0Kh4hlFdgGRASX71+3D3twREUIXQsPxUDBCPVhIaQtRCAkY0fiWFloDIrHxZ6qIAyawlNAIsAWEJjfMBYg0Ny4+EhhD1RgJGOEZJQZUb3jUFiI2hEdrnwmOv4CoBEgQNMAmTEKIyCRhRfyqFRtUWUxUDxIbQ8AquEhqtKweIhIYQjZ4EjKjd+dCwpcltaUH1+3APuNBK6nxoVLw0dT5APFtJaAjRjEjAtEQlBVVaSdXS5Lam0PAIvHDDO7RP5Wa2FQNEQkOIFksCprkozq+lmW2VG+K2hEZY31r6aUhoCCEuTQKmMasYGtU1s71kaCjwCLhwVlExNC7qpyGhIYSoXxIwDc0aGjU0s614qaq0sJodWELjfCspa2jU0E9DxnESQjiIHH3qQ3F+7c1sK7aiqjE0Ai/c8A7rW3s/DQkNIUQTIEequtj6LuxfdiFAbAmNdv2r9NOoEBwSGkKIZkiOanWhTeDiWSU0qrSiktAQQrRwcgSsi0EPmX+EEELUSCZ8EEIIYRcSMEIIIexCAkYIIYRdSMAIIYSwC7sGjFJqrFLqoFIqWSn1VDXLXZVSiy3L/6uUCq+wLFIptVUptV8p9YtSyq3KtiuUUvsqPA9QSq1RSh2y/Pa352cTQghRO7sFjFLKCLwDxALXAtOUUtdWWe1eIEdr3Ql4HXjZsq0T8Clwv9a6GzAcKK2w70lA1fHenwLWaa07A+ssz4UQQjiIPc9g+gHJWusjWusS4AtgQpV1JgAfWx4nACOVUgoYDezVWu8B0Fpnaa3LAZRSXsDjwOxa9vUxcFM9fx4hhBCXwZ4BEwocr/A81fJatQANW4cAAAgWSURBVOtorcuAPCAQ6AJopdQqpdQupdQTFbb5F/AaULX7fGut9QnLvk4AwdUVpZSaqZTaoZTakZmZWbdPJoQQ4pLs2dFSVfOatnEdJ2AI0BdzkKxTSu0EsoBOWuvHKt6vuRxa6/eB9wGUUplKqd/rsh+gFXC6jtvak9R1eaSuyyN1XZ7GWhdcWW3tbVnJngGTCrSr8DwMSK9hnVTLfRdfINvy+kat9WkApVQi0BvzfZc+SqmjltqDlVLfa62HA6eUUm211ieUUm2BjEsVqLUOquuHU0rt0FpfV9ft7UXqujxS1+WRui5PY60LGqY2e14i2w50VkpFKKVcgKnAiirrrADusjyOB9ZrrTWwCohUSnlYgmcYcEBr/Z7WOkRrHY75DOc3S7hU3dddwNd2+lxCCCFsYLczGK11mVLqIcxhYQQWaK33K6X+CezQWq8APgAWKqWSMZ+5TLVsm6OUmoM5pDSQqLX+7hJv+RLwpVLqXuAYMMUuH0wIIYRN7DrYpdY6EUis8trfKzwuooYg0Pr/t3evIXZVZxjH/482YKrBW3pRtOaDotZbTEtIiQyiflBbItaAQW0ZUXqT1iJSsBSDLVgRvFWxiYptxHjDqmjEkjRoxYi3TNNEm2ALDaUqhEqrphXB5O2HtY49Hs+Z7JOZtdepPj84ZM9ea5/95p3Zs+asvfe7427SpcqD3nsrcGzX128Cp04t4qHc1uK+huG4huO4huO4hjOqcUELsSnNSJmZmU0vl4oxM7MiPMBMQtKdkrZ1l6TpaZekX+RSNxslzRuRuE6W9JakDfl1Zb9+BeI6VNKTkjbnEj+X9unTes4axtV6ziTtJekFSX/McV3Vp8/AckqV4xrPl/l38nVx6bi69r2npD9IWtWnrfV8NYyrSr4kbVUqtbVB0kt92ssejxHh14AXMEa6PPrlAe1nAk+Q7udZADw/InGdDKyqkK+DgHl5eRbwKvDF2jlrGFfrOcs52CcvzwCeBxb09PkesCwvLwHuH5G4xoFb2v4Zy/u+DLin3/erRr4axlUlX8BWYPYk7UWPR3+CmUREPE26um2Qs4C7InkO2C/fg1M7rioi4o2ImMjL7wCb+Wj1htZz1jCu1uUcdGrqzciv3pOig8op1Y6rCkmHAF8F7hjQpfV8NYxrVBU9Hj3ATE2Tcji1fCVPcTwh6Zi2d56nJk4k/fXbrWrOJokLKuQsT6tsIN0YvCYiBuYrPlxOqXZcAOfkaZUHJR3ap72EG4EfATsHtFfJV4O4oE6+Algtab2kb/VpL3o8eoCZmiblcGqYAA6LiBOAm4FH2ty5UkHS3wA/jIi3e5v7bNJKznYRV5WcRcSOiJhLqnQxX9KxPV2q5KtBXI8BcyLieOB3/O9TQzGSvgZsi4j1k3Xrs65ovhrG1Xq+soURMY9U1f4SSWM97UXz5QFmapqUw2ldRLzdmeKIdC/SDEmz29i3pBmkX+IrI+KhPl2q5GxXcdXMWd7nv4CngNN7mj7Ilz5cTqlqXJEqnL+Xv7wd+FIL4SwEFimViroPOEVS771yNfK1y7gq5YuIeD3/uw14mFTlvlvR49EDzNQ8CnwzX4mxAHgrckXnmiR9vjPvLGk+6fv8Zgv7Fak6w+aIuH5At9Zz1iSuGjmT9BlJ++XlmcBpwJaeboPKKVWNq2eefhHpvFZREXFFRBwSqVTUElIuLujp1nq+msRVI1+S9pY0q7NMegxK75WnRY/Honfy/7+TdC/p6qLZkv4OLCWd8CQilpGqFJwJ/IVU9fnCEYlrMfBdSe8D7wJLSh9k2ULgG8CmPH8P8GPgC12x1chZk7hq5OwgYIXSw/n2AB6IiFVqUE5pBOL6gaRFwPs5rvEW4uprBPLVJK4a+foc8HD+u+lTwD0R8VtJ34F2jkffyW9mZkV4iszMzIrwAGNmZkV4gDEzsyI8wJiZWREeYMzMrAgPMGZmVoQHGLMRl0uu71ZVgVwm/uDpeC+zYXmAMft4GwcO3lUnsxI8wJg1JGmOpC2S7pD0sqSVkk6TtE7SnyXNz69nlR489aykI/O2l0m6My8fl7f/9ID9HChpdX6P5XQVJJR0gdLDwDZIWp7vtkfSdknXSZqQtDaXe1kMfBlYmfvPzG/z/dxvk6SjSubMPtk8wJgN53DgJuB44CjgPOAk4HJS+ZktwFhEnAhcCVydt7sROFzS2cCvgG9HxH8G7GMp8Ex+j0fJJW0kHQ2cS6qQOxfYAZyft9kbmMiVc38PLI2IB4GXgPMjYm5EvJv7/iP3+2WO26wI1yIzG85fI2ITgKRXgLUREZI2AXNI1XtXSDqCVPa8UyNup6RxYCOwPCLWTbKPMeDrebvHJf0zrz+VVIX3xVxfaibpeS2QnkNyf16+G+hXybqj07a+sx+zEjzAmA3nva7lnV1f7yQdTz8DnoyIs5UebvZUV/8jgO00OyfSr0iggBURccVubt/RiXkH/h1gBXmKzGx67Qu8lpfHOysl7UuaWhsDDsznRwZ5mjz1JekMYP+8fi2wWNJnc9sBkg7LbXuQKkJDmrZ7Ji+/A8yawv/HbLd5gDGbXtcCP5e0Dtiza/0NwK0R8SpwEXBNZ6Do4ypgTNIE6RkefwOIiD8BPyE9AncjsIZUWh/g38AxktYDpwA/zet/DSzrOclv1gqX6zf7GJC0PSL2qR2HWTd/gjEzsyL8CcasEkkXApf2rF4XEZfUiMdsunmAMTOzIjxFZmZmRXiAMTOzIjzAmJlZER5gzMysCA8wZmZWxH8BTKx4RWHqALAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize results\n",
    "print(\"Best: %f using %s\" % (gsearch2_1.best_score_, gsearch2_1.best_params_))\n",
    "test_means = gsearch2_1.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = gsearch2_1.cv_results_[ 'std_test_score' ]\n",
    "train_means = gsearch2_1.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = gsearch2_1.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "pd.DataFrame(gsearch2_1.cv_results_).to_csv('my_preds_maxdepth_min_child_weights_1.csv')\n",
    "\n",
    "# plot results\n",
    "test_scores = np.array(test_means).reshape(len(max_depth), len(min_child_weight))\n",
    "train_scores = np.array(train_means).reshape(len(max_depth), len(min_child_weight))\n",
    "\n",
    "for i, value in enumerate(max_depth):\n",
    "    pyplot.plot(min_child_weight, -test_scores[i], label= 'test_max_depth:'   + str(value))\n",
    "#for i, value in enumerate(min_child_weight):\n",
    "#    pyplot.plot(max_depth, train_scores[i], label= 'train_min_child_weight:'   + str(value))\n",
    "    \n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'max_depth' )                                                                                                      \n",
    "pyplot.ylabel( 'Log Loss' )\n",
    "pyplot.savefig('max_depth_vs_min_child_weght_1.png' )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'max_depth': [4, 5, 6], 'min_child_weight': [0, 1, 2]}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max_depth = [4,5,6]\n",
    "min_child_weight = [0,1,2]\n",
    "param_test2_2 = dict(max_depth=max_depth, min_child_weight=min_child_weight)\n",
    "param_test2_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ice2018/anaconda2/lib/python2.7/site-packages/sklearn/model_selection/_search.py:762: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: -0.06416, std: 0.00090, params: {'max_depth': 4, 'min_child_weight': 0},\n",
       "  mean: -0.06417, std: 0.00106, params: {'max_depth': 4, 'min_child_weight': 1},\n",
       "  mean: -0.06422, std: 0.00106, params: {'max_depth': 4, 'min_child_weight': 2},\n",
       "  mean: -0.06404, std: 0.00108, params: {'max_depth': 5, 'min_child_weight': 0},\n",
       "  mean: -0.06404, std: 0.00119, params: {'max_depth': 5, 'min_child_weight': 1},\n",
       "  mean: -0.06399, std: 0.00113, params: {'max_depth': 5, 'min_child_weight': 2},\n",
       "  mean: -0.06421, std: 0.00115, params: {'max_depth': 6, 'min_child_weight': 0},\n",
       "  mean: -0.06413, std: 0.00109, params: {'max_depth': 6, 'min_child_weight': 1},\n",
       "  mean: -0.06415, std: 0.00123, params: {'max_depth': 6, 'min_child_weight': 2}],\n",
       " {'max_depth': 5, 'min_child_weight': 2},\n",
       " -0.06398936452777433)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb2_2 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=93,  #第一轮参数调整得到的n_estimators最优值\n",
    "        max_depth=5,\n",
    "        min_child_weight=1,\n",
    "        gamma=0,\n",
    "        subsample=0.3,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel = 0.7,\n",
    "        objective= 'binary:logistic',\n",
    "        seed=3)\n",
    "\n",
    "\n",
    "gsearch2_2 = GridSearchCV(xgb2_2, param_grid = param_test2_2, scoring='neg_log_loss',n_jobs=-1, cv=kfold)\n",
    "gsearch2_2.fit(X_train , y_train)\n",
    "\n",
    "gsearch2_2.grid_scores_, gsearch2_2.best_params_,     gsearch2_2.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best: -0.063989 using {'max_depth': 5, 'min_child_weight': 2}\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAacAAAELCAYAAAB9MUkNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8FeXZ+P/PlT2QAFkIW9iXBBfWAKKCLIIb4oaCra1WrW3V0kd/WrHaqtX2QUst9evTxS7W2tYAUpUissgiikUEBRdI2IUAJmQBEkhCluv3x0zCSThZyck5Sa7365VXzszcM3Od4ZDr3DP3IqqKMcYYE0iC/B2AMcYYU50lJ2OMMQHHkpMxxpiAY8nJGGNMwLHkZIwxJuBYcjLGGBNwLDkZY4wJOJacjDHGBBxLTsYYYwJOiL8DaKni4+O1T58+/g7DGGNajPj4eFasWLFCVa+sq6wlp0bq06cPmzdv9ncYxhjToohIfH3K2W09Y4wxAceSkzHGmIBjyckYY0zAsWdOTaikpISMjAyKior8HYpp5SIiIkhMTCQ0NNTfoRjjE5acmlBGRgbR0dH06dMHEfF3OKaVUlVycnLIyMigb9++/g7HGJ+w23pNqKioiLi4OEtMxqdEhLi4OKuhm1bNklMTs8RkmoN9zkxrZ8nJGGNM3crL4dAW2PiHZjmdPXMyxhjjXelp2P8+pL0N6csg/whIMFxwI0Ql+PTUVnNqRY4dO8bvfve7Ru07f/58Tp06dc4xbN68mdmzZ5/zcSrccccdvP7662etP3z4MDNmzABg3bp1TJs2zev+ffr0ITs7u8niqXD33Xezffv2WsvUFPv+/fv517/+VeN+y5cvJykpiQEDBjB37txzjtWYBik6AV8shtfvhF/1h3/cCNteg8QUuP4P8PBunycmsJpTq1KRnO69994G7zt//nxuu+022rVrd04xpKSkkJKSck7HqI/u3bt7/cPfXP785z83et+K5PSNb3zjrG1lZWXcd999rFq1isTEREaNGsX06dM577zzziVcY2p34ohTM0p7G/ath/ISaBcP510HyddAvwkQGtmsIVly8pGn/vMl2w+faNJjnte9A09ce36N2+fMmcOePXsYNmwYU6ZMISEhgYULF1JcXMwNN9zAU089xcmTJ7nlllvIyMigrKyMn/70p2RmZnL48GEmTpxIfHw8a9eu9Xr8qKgo7rvvPt59911iYmL45S9/yY9//GMOHDjA/PnzmT59OuvWrWPevHksXbqUJ598kgMHDrB3714OHDjA//zP/9Raq/r73//OvHnzEBGGDBnCq6++CsD69et5/vnn+frrr3nuueeYMWMG+/fvZ9q0aXzxxRdVjpGTk8Ott97K0aNHGT16NKpa4/mee+45IiIimD17Ng888ADbtm1jzZo1rF69mpdffpl//OMfrFy5kieeeILi4mL69+/Pyy+/TFRUFBMmTGDevHmkpKTwl7/8hWeffZbu3bszcOBAwsPDefHFF2uMfc6cOezYsYNhw4Zx++2388ADD1TGtGnTJgYMGEC/fv0AmDVrFm+99ZYlJ9O0VOFoOqS/7SSkQ1uc9TF94aLvQ9I10HM0BAX7LUS7rdeKzJ07l/79+7N161amTJnCrl272LRpE1u3bmXLli2sX7+e5cuX0717d7Zt28YXX3zBlVdeyezZs+nevTtr166tMTEBnDx5kgkTJrBlyxaio6N5/PHHWbVqFW+88QY/+9nPvO6TlpbGihUr2LRpE0899RQlJSVey3355Zf84he/YM2aNWzbto3f/va3lduOHDnCBx98wNKlS5kzZ06t1+Cpp57i0ksv5dNPP2X69OkcOHCgxrLjx4/n/fffB5zbkQUFBZSUlPDBBx8wbtw4srOzeeaZZ3j33Xf55JNPSElJ4fnnn69yjMOHD/P000+zceNGVq1aRVpaWpXt3mKfO3cu48aNY+vWrTzwwAMcPnyYq6++GoBDhw7Rs2fPyv0TExM5dOhQre/ZmHopL4MDG2HlT+H/jYTfjYHVP3cS1aTH4d6NMPtTmPoM9B7r18QEVnPymdpqOM1h5cqVrFy5kuHDhwNQUFDArl27GDduHA899BCPPPII06ZNY9y4cfU+ZlhYGFde6Yx0f+GFFxIeHk5oaCgXXngh+/fv97rPNddcQ3h4OOHh4SQkJJCZmUliYuJZ5dasWcOMGTOIj3cGLI6Nja3cdv311xMUFMR5551HZmZmrTGuX7+ef//735XnjomJqbHsyJEj2bJlC/n5+YSHhzNixAg2b97M+++/zwsvvMDGjRvZvn07l1xyCQCnT59m7NixVY6xadMmLrvsssp4b775Znbu3Nmg2Lt3786yZcsAvNb0rNm4abSSIti7zqkhpb8DJ49CUAj0HQ8X/QCSroaOPfwdpVd+S04iEgssAPoA+4FbVDXPS7nbgcfdxWdU9RV3fRjwIjABKAceU9XFHvvNABYBo1R1s7vuUeAuoAyYraor3PVXAr8FgoE/q2qLfwqtqjz66KN873vfO2vbli1bWLZsGY8++ihTp06tsdZTXWhoaOUfyqCgIMLDwytfl5aWet2nogxAcHBwjeVUtcY/wp7HqO02XYX6/jEPDQ2lT58+vPzyy1x88cUMGTKEtWvXsmfPHgYPHsyePXuYMmUKr732Wo3HqCuehsaemJjIwYMHK5czMjLo3r17Pd6NMa5TubBrpXO7bvdqKDkJYdEwcIrz/GjA5RDZyd9R1smft/XmAKtVdSCw2l2uwk1gTwBjgNHAEyJS8VX4MSBLVQcB5wHveewXDcwGPvJYdx4wCzgfuBL4nYgEi0gw8H/AVe5xbnXLtjjR0dHk5+cDcMUVV/DXv/6VgoICwLldlJWVxeHDh2nXrh233XYbDz30EJ988slZ+/rD5MmTWbhwITk5OQDk5uY26jjjx4/nn//8JwDvvPMOeXlnfd85q/y8efMYP34848aN4w9/+APDhg1DRLjooovYsGEDu3fvBuDUqVNVakUAo0eP5r333iMvL4/S0lIWL17s7TRV1HatR40axa5du9i3bx+nT58mNTWV6dOn1+etm7bs2AGn/9Er18KvBsAb34ODm2DoTPjmYvjxHrj5ZbhwRotITODf23rX4dR6AF4B1gGPVCtzBbBKVXMBRGQVTmJ5DbgTSAZQ1XLAs73w08BzwEPVzpeqqsXAPhHZjZPwAHar6l73HKlu2drbCQeguLg4LrnkEi644AKuuuoqvvGNb1TehoqKiuIf//gHu3fv5uGHHyYoKIjQ0FB+//vfA3DPPfdw1VVX0a1bt1qfO/nK+eefz2OPPcZll11GcHAww4cP529/+1uDj/PEE09w6623MmLECC677DJ69epVa/lx48bxi1/8grFjx9K+fXsiIiIqb3V27tyZv/3tb9x6660UFxcD8MwzzzBo0KDK/Xv06MFPfvITxowZQ/fu3TnvvPPo2LFjreccMmQIISEhDB06lDvuuIOZM2dy9913s2zZMkJCQnjxxRe54oorKCsr48477+T88/17i9gEIFXI/ALSlkHaUvj6M2d9fBJc8iNIngbdh0NQy21WIPW51eCTE4scU9VOHst5qhpTrcxDQISqPuMu/xQoBP4MfI5z224CsAe4X1UzRWQ48Liq3iQi64CHVHWziLwIbFTVf7jH+gvwjnuqK1X1bnf9t4Axqnp/bfGnpKRo9Zlwd+zYweDBgxtxNUxLVlBQQFRUFKWlpdxwww3ceeed3HDDDT4/r33e2piyUjjwX7fJ91KntoQ4reqSr3Fa2MUP8HeUdRKRLapaZ38Tn9acRORdoKuXTY/V9xBe1ilO3InABlV9UEQeBOa5z6d+A9zRgGN5+2rhNWOLyD3APUCd38hN2/Hkk0/y7rvvUlRUxNSpU7n++uv9HZJpLU6fhD1rnBrSznegMA+Cw51+R+MegqSrmqVDrD/4NDmp6uU1bRORTBHppqpHRKQbkOWlWAZnbv2Bk5DWATnAKeANd/0inIYO0cAFwDr3oXhXYImITHeP1bPasQ67r2taX/39vAS8BE7Nqab31tKNGTOm8jZWhVdffZULL7zwnI+dk5PD5MmTz1q/evVq4uLizvn4/jjnvHnzzvkYxlQ6me20rEtf5iSm0iKI6AiDrnRqSP0nQ3iUv6P0OX8+c1oC3A7MdX+/5aXMCuCXHo0gpgKPqqqKyH9wEtcaYDKwXVWPA/EVO1e7rVcI/EtEnge6AwOBTTg1qoEi0hc4hNNo4uyu+23IRx99VHehRoqLi2Pr1q0+O36gnNOYBsnd67SuS1sGBzeClkOHRBhxu5OQel8MwW1rYkl/Jqe5wEIRuQs4ANwMICIpwPdV9W5VzRWRp4GP3X1+XtE4AqfxxKsiMh84CnyntpOp6pcishCnoUMpcJ+qlrnnvB8nEQYDf1XVL5vyjRpjTBWqcPjTMwOqZrntr7pcAOMfdhJS1yHQhvu4+a1BREtnDSKMv9nnrYUpPQ1ffeAmpHfgxCGQIOh1sZOMkq+GmD7+jtLnAqJBhDHGtGlFJ2D3u05C2rUKio9DSCQMmOwMGTTwCmjvm2etLZ0lJ2OMaUr5X1cd4bvsNLSLg8HXnhnhO+zcRv9vC1puDy1zFpvP6WwtcT6nO++8k4SEBC644IJzjtM0k6M74f3n4U+T4ddJsPQByNkNo++B77wDD+2C6//PuXVnialeLDm1IoGQnFJSUnjhhRfO+Th1CYT5nBo7jUVdyemOO+5g+fLljQ3NNIfycmd4oFU/c0b4/r9RsPop0DKY+Dj84L8weytc8QunpZ2fR/huiey2nq+8Mwe+/rxpj9n1Qriq5jFpbT6nlj+fEzjj/dU0yrvxo5Ii5zZd2lJ3hO8sZ4TvPuNgzPedDrEdzx5x3zSO1ZxaEZvPqeXP52QCTGEefLYQFn7bmbL8Xzc7U5j3vhhu/DM8vAe+/SaM/q4lpiZmNSdfqaWG0xxsPqeWOZ+TCQDHM84MqPrVBigvhaguzojeydOcuZBCwus+jjknlpxaKZvPqW6BOJ+T8QNVpxNs2ttOQjqyzVkfPwjG3u8kpB4jW/QI3y2RXe1WxOZzavnzOZlmUlYK+z+A5T+B3w6F318Ma3/hDKp6+ZNw38dw/8cw5SnoOcoSkx9YzakVsfmcWv58TgC33nor69atIzs7m8TERJ566inuuuuuBl8LU83pU7B37ZkRGgpzITjM6Xd06QNOg4Zob5MoGH+w4YsayYYvMhVsPqcAdjIHdi53EtKeNVBaCOEdYdAVTp+jAZdDeLS/o2xTbPgiY5qJzecUYHL3nRmh4cB/3RG+e8CIb0HS1dDn0jY3wndLZMnJnMXmc2oYm8/Jz1ThyFa3hd3bkOVOKpBwvjMhX/LV0G1Ymx7huyWy5GTOYvM5mYBXVuI0aEhf5iSlExnuCN9j4YpfOjWk2L7+jtKcA0tOxpiWoTjfHeF7GexaAUXuCN/9J8HEnzjPkdrH130c0yJYcjLGBK78TKd2lL4M9q5zRviOjHX6HiVfA/0m2kCqrZQlJ2NMYMne5XSGTVsGGR8DCp16w6jvOs+Pel4Ewfanq7WznmWtSCCMSm5TZpzRmCkzDh48yMSJExk8eDDnn39+lTEGW63ycjj4Max6Al4cBS+mwLtPQlmxc7vuBx/Cj7bBlb90W9pZYmoL7F+5FalITvfee2+D950/fz633XYb7dqd2y2SlJQUUlLq7MJwzgJhyozGqkhO3/jGN87aFhISwq9//WtGjBhBfn4+I0eOZMqUKY2eniNglRZXHeG7IBMk2Ek+o77rdIjt1NPfURo/sppTK+I5ZcbDDz/Mr371K0aNGsWQIUN44oknAGdk8WuuuYahQ4dywQUXsGDBAl544YXKKTMmTpxY4/GjoqJ45JFHGDlyJJdffjmbNm1iwoQJ9OvXjyVLlgBVazFPPvkkd955Z2WZuuZ5+vvf/86QIUMYOnQo3/rWtyrXr1+/nosvvph+/fpVJqT9+/d7nYwvJyeHqVOnMnz4cL73ve/VOWVGRUwPPPAAkyZNApwm5rfddhvgDKA7duxYRowYwc0331w5HNSECROo6IT9l7/8hUGDBjFhwgS++93vcv/999ca+5w5c3j//fcZNmwYv/nNb6rE1K1bN0aMGAE4wxwNHjyYQ4cO1XrdWozCY/DZIlh4OzzXD/45w1nudRHc+Cf48R64fQmMuccSk7Gak688u+lZ0nLT6i7YAMmxyTwy+pEat8+dO5cvvviCrVu3snLlSl5//XU2bdqEqjJ9+nTWr1/P0aNH6d69O2+//TYAx48fp2PHjjz//POsXbu2clRwbyqmzHj22We54YYbKqfM2L59O7fffjvTp08/a5+0tDTWrl1Lfn4+SUlJ/OAHPyA09OwOkBVTZmzYsIH4+PgqY+tVTDuRlpbG9OnTK2/neVMxZcbPfvYz3n77bV566aUay44fP55f//rXzJ49m82bN1NcXFzjlBnt27fn2Wef5fnnn68yUG7FlBmffPIJ0dHRTJo0iaFDh9Ya+9y5cyvnvKo4hufwRRX279/Pp59+ypgxY2p8DwHv+CG3ufdSp+l3eSm0T4ALbjozwndohL+jNAHIklMrZVNmtOwpMwoKCrjpppuYP38+HTp0qPU9BxRVyNrhjl/3Nhz+1FkfNwDG3ueO8J1iA6maOlly8pHaajjNwabMqFugTplRUlLCTTfdxDe/+U1uvPHGeu3jV+VlcPCjM1NO5O131vdIgclPOAmp86BaD2FMdfb1pRWxKTNa/pQZqspdd93F4MGDefDBB+vzlv2jpNBp6v3mfTBvILx8FWx6yakhTfsNPJgG310N4x60xGQaxWpOrYhNmdHyp8zYsGFD5TiGw4YNA+CXv/xlYEzjfiq36gjfJaecEb4HTnE6xA64HCJa0C1IE9BsyoxGsikzTIVWPWVG3v4zA6oe+NAZ4Tu6u9MZNvka6H0phIT5NgbTqtiUGcY0k1Y1ZYaqM015xZQTmV846xPOg0sfdBJS9+E2wrfxOUtO5iw2ZUbDtPgpM8pK4KsP3RZ2y+D4QUCcEb6nPuOM8B3X399RmjbGkpM5i02Z0QYUF8Ce1U5C2rncHeE7whnh+7JHYNCVENXZ31GaNsySUxOrrUm0MU2lUc+KC7KcoYLS3nZH+C6GyBinZpR8jZOYwto3eazGNIYlpyYUERFBTk4OcXFxNSeo/K+dXvIACIj7u0HLFeuqb8d9FlDT/ng8K5CzlxsVSw3LlqB9RlXJyckhIqIeIytk73Y6w6a9DQc34Yzw3QtG3eUkpV5jbSBVE5DsU9mEEhMTycjI4OjRozUXys+E8pIzy1W+Abe2lpPi8at6sqopiUmVX1X3q+N4Xvepfg6p9rKm4zfFtnM4XpW3cPaXi4jQYBLjopwvOxLkDJoqAkHBTkJKW+okpOx0Z7+uQ2DCo04ruy4X2JcHE/AsOTWh0NBQ+vata2roWpr+qro/ZU6T3XL3d+VyeR3b3OXatlXuW9u28hrO4bHtrON421Y9npq2NeY9aw3n8LKtzvdVy7VrqSQY+lzi1pCucmpLxrQglpwCiYj7jdYG7ggYjUrsZbVvq0yeDUm69fjCUPE6KsHpENsutu73Z0yA8ktyEpFYYAHQB9gP3KKqZ40zIyK3A4+7i8+o6ivu+jDgRWACUA48pqqLPfabASwCRqnqZnfdo8BdQBkwW1VXuOv3A/nu+tL6dA4zbUhQEM6XBfseZ0xz8tf/uDnAalWdKyJz3OUqI6W6CewJIAXnYcwWEVniJrHHgCxVHSQiQUCsx37RwGzgI4915wGzgPOB7sC7IjJIVcvcIhNVtemnSzXGGNMo/rp/dB3wivv6FcBbl/orgFWqmusmpFXAle62O4H/BVDV8mqJ5WngOaCo2vlSVbVYVfcBu4HRTfVmjDHGNC1/JacuqnoEwP2d4KVMD+Cgx3IG0ENEOrnLT4vIJyKySES6AIjIcKCnqi6tz7Hc1wqsFJEtInLPOb0rY4wxTcJnt/VE5F2gq5dNj9X3EF7WKU7MicAGVX1QRB4E5rnPp34D3NGAYwFcoqqHRSQBWCUiaaq63mtATvK6B6hztGtjjDGN57PkpKqX17RNRDJFpJuqHhGRbkCWl2IZOA0eKiQC64Ac4BTwhrt+EU5Dh2jgAmCd2wG2K7BERKa7x+pZ7ViH3TgrfmeJyBs4t/u8JidVfQl4CZxRyWt6f8YYY86Nv27rLQFud1/fDrzlpcwKYKqIxIhIDDAVWKHOuC3/4UzimgxsV9Xjqhqvqn1UtQ+wEZjuttZbAswSkXAR6QsMBDaJSHu3AQUi0t49xxc+eL/GGGMawF+t9eYCC0XkLuAAcDOAiKQA31fVu1U1V0SeBj529/m5qlZMj/oI8KqIzAeOAt+p7WSq+qWILAS2A6XAfapa5j6resOtaYUA/1LV5U36To0xxjSYTTbYSN4mGzTGGFO7+k42aEMRGGOMCTiWnIwxxgQcS07GGGMCjiUnY4wxAceSkzHGmIBjyckYY0zAseRkjDEm4FhyMsYYE3AsORljjAk4lpyMMcYEHEtOxhhjAo4lJ2OMMQHHkpMxxpiAY8nJGGNMwLHkZIwxJuBYcjLGGBNwLDkZY4wJOA1KTiISIyJDfBWMMcYYA/VITiKyTkQ6iEgssA14WUSe931oxhhj2qr61Jw6quoJ4EbgZVUdCVzu27CMMca0ZfVJTiEi0g24BVjq43iMMcaYeiWnnwMrgN2q+rGI9AN2+TYsY4wxbVlIXQVUdRGwyGN5L3CTL4MyxhjTttWnQcRzboOIUBFZLSLZInJbcwRnjDGmbarPbb2pboOIaUAGMAh42KdRGWOMadPqk5xC3d9XA6+paq4P4zHGGGPqfuYE/EdE0oBC4F4R6QwU+TYsY4wxbVmdNSdVnQOMBVJUtQQ4CVzn68CMMca0XXXWnEQkFPgWMF5EAN4D/uDjuIwxxrRh9bmt93uc506/c5e/5a6721dBGWOMadvqk5xGqepQj+U1IrLNVwEZY4wx9WmtVyYi/SsW3BEiynwXkjHGmLauPjWnh4G1IrIXEKA38B2fRmWMMaZNq8/wRatFZCCQhJOc0oBhvg7MGGNM21WvyQZVtVhVP1PVbapajMdYe40hIrEiskpEdrm/Y2ood7tbZpeI3O6xPkxEXhKRnSKSJiI3VdtvhoioiKS4y3EislZECkTkxWplR4rI5yKyW0ReELdJojHGGP9p7DTt5/oHfA6wWlUHAqvd5aoncCY3fAIYA4wGnvBIYo8BWao6CDgPp3l7xX7RwGzgI4/DFQE/BR7yEsvvgXuAge7Plef0zowxxpyzxiYnPcfzXge84r5+BbjeS5krgFWqmquqecAqziSOO4H/BVDVclXN9tjvaeA5PEaxUNWTqvoB1Ua2cOep6qCq/1VVBf5eQyzGGGOaUY3PnETkP3hPQgLEneN5u6jqEQBVPSIiCV7K9AAOeixnAD1EpJO7/LSITAD2APeraqaIDAd6qupSEfFWS/J2jozq52jgezHGGNPEamsQMa+R2wAQkXeBrl42PVbXvhWH8LJOcWJOBDao6oMi8iAwz30m9Rvgjnoev7ZzeC8scg/OLUB69erVgNMYY4xpiBqTk6q+V9O2+lDVy2vaJiKZItLNrTV1A7K8FMsAJngsJwLrgBzgFPCGu34RcBcQDVwArHPbNHQFlojIdFXdXEMoGe5xPc9xuJb39BLwEkBKSsq53to0xhhTg8Y+czpXS4CK1ne3A295KbMCmCoiMW5DiKnACvfZ0H84k7gmA9tV9biqxqtqH1XtA2wEaktMuLcW80XkIreV3rdriMUYY0wzqk8nXF+YCywUkbuAA8DNAG7T7++r6t2qmisiTwMfu/v83GMuqUeAV0VkPnCUenQKFpH9QAcgTESux5lEcTvwA+BvQCTwjvtjjDHGj8SpiJiGSklJ0c2ba6yUGWOM8UJEtqhqSl3l6jNlhrdWe8eBzcAfVdUmHjTGGNOk6vPMaS9QAPzJ/TkBZAKD3GVjjDGmSdXnmdNwVR3vsfwfEVmvquNF5EtfBWaMMabtqk/NqbOIVHbqcV/Hu4unfRKVMcaYNq0+Naf/D/hARPbgdFrtC9wrIu05MwSRMcYY02TqM2XGMnfKjGTcKTM8GkHM92Vwxhhj2qb6tNYLBb4HVDx3Wicif1TVEp9GZowxps2qz2293wOhwO/c5W+56+72VVDGGGPatvokp1GqOtRjeY2IbPNVQMYYY0x9WuuViUj/igUR6QeU+S4kY4wxbV19ak4PA2tFZC9Og4je1GMsO2OMMaax6tNab7XbWi8Jt7UeMMzXgRljjGm76jUquaoWA59VLIvIIsBm2zPGGOMTjZ3PydsMssYYY0yTaGxysnk2jDHG+EyNt/VqmCoDnFpTnM8iMsYY0+bV9sxpXiO3GWOMMeekxuSkqu81ZyDGGGNMhcY+czLGGGN8xpKTMcaYgNOg5CQiXX0ViDHGGFOhoTWnZT6JwhhjjPFQrxEiPFjnW2MCWFpuGvuO7yMmIoaY8BjiIuPoFN6JkKCG/lc3xr8a+on9k0+iMMY0WnFZMSv3ryQ1PZXPjn7mtUyn8E7ERsQSGxFLTEQMsRGxxEXEOesiz6yPi4gjOiyaILHH0ca/GpScVPV3dZcyxjSHQwWHWJi+kDd2vUFecR59OvThkVGPMLrbaI4XHyevKI/cotwqPzmFOew+tpvcolyOFx/3etwQCalMYFWSWWSc1wQXGRKJiN1UMU3L6vrGtCDlWs6GQxtYkL6A9RnrEREm9pzIzKSZXNTtogYliZLyEo4VHTsrgVX+FOaSW5zLwaMHyS3K5VTpKa/HiQiOOCtpxUZ61Mw810fEEhYc1lSXw7RilpyMaQGOFR3jzd1vsiB9ARkFGcRFxPHdId/l5kE307V94xrRhgaF0rldZzq361yv8oWlhV5rY7mFZ15nF2azM28nuUW5lJSXeD1OdGj0mVuJ4TGVrytqYp6JrFN4J4KDghv1/kzLZsnJmAD2ZfaXvJb2Gsv3L6e4rJgRCSOYPWI2l/e6nNDg0GaNJTIkksioSLpHda+zrKpSUFJQayLLK8rjQP4Bth7dyrHiY5Rr+VlSwvR1AAAfaElEQVTHEaQyWXkmLa/JLDKW6NBou8XYSlhyMibAFJUWsWL/ClLTUvki5wsiQyK5rv913JJ0C0mxSf4Or15EhOiwaKLDoundoXed5cvKyzh++ji5hbnkFeeRU5RTJZFVJLP03HRyinLIP53v9TghQSFea2Bn/bi1tciQyKZ+66aJWHIyJkAczD/oNHDY/QbHi4/Tr2M/Hh39KNf2v5bosGh/h+dTwUHBlYmjPkrKSqokrZyinLNqaXlFeXx14ityCnMoKivyepzIkMjaE1h41daMoUHNW1ttyyw5GeNHZeVlbDi8gdfSXmPDoQ0ESRCTek1iVtIsRnUdZbeoahAaHEqX9l3o0r5LvcqfKjlVmbAqWy56JLS8ojwyT2WyI2cHuUW5lGqp1+N0COtQYy2s+k/H8I7WJP8cWHIyxg/yivL4965/s2jnIg4VHKJzZGe+P/T73DTwpnr/wTX11y60He1C25EYnVhnWVXlxOkT3pOZe6sxrziPfcf3sSVzC8eKj6Fepr4LkiBiwmMq+49VT2RV1kfE0j60vX0Z8WDJyZhmoqp8nv05qWmprNi/gtPlp0npksIDIx9gUq9JdssoQIgIHcM70jG8I3079q2zfGl5KceKj52dzApzqqz7MudLcotyKSgp8HqcsKAwYiPPtGD0TFzVk1lMRAwRIRFN/dYDiiUnY3yssLSQ5fuW81raa+zI3UG7kHbcOPBGZibNZEDMAH+HZ85RSFAI8ZHxxEfG16t8cVnx2U3yC8/Uziq27T22l5zCHE6Xn/Z6nPah7as0xa+pX1lLHcKqZUVrTAvy1YmvWJC+gDd3v0n+6XwGdBrA42MeZ1r/abQPbe/v8IyfhAeH07V913r1T1NVTpWequwQXb0FY8XP4YLDfJH9BXlFeZRpmddjdQrvdFYLRs+amGfn6UAYwsovyUlEYoEFQB9gP3CLquZ5KXc78Li7+IyqvuKuDwNeBCYA5cBjqrrYY78ZwCJglKpuFpE44HVgFPA3Vb3fo+w6oBtQ6K6aqqpZTfVeTdtSVl7G+oz1pKan8uHhDwmRECb3nsyspFmM7DLSnimYBhER2oe2p31oe3rSs87y5VrOieITtSay3KLceg9hdVYyi4wjJjyGa/tf6/ORPvxVc5oDrFbVuSIyx11+xLOAm8CeAFIABbaIyBI3iT0GZKnqIBEJAmI99osGZgMfeRyuCPgpcIH7U903VXVzk7070+bkFObwxu43WJi+kCMnj5AQmcC9w+5lxsAZ9R6BwZhzFSRBdIroRKeITvTr2K/O8nUOYeX+ZBzNqDKE1fT+0339VvyWnK7DqfUAvAKso1pyAq4AVqlqLoCIrAKuBF4D7gSSAVS1HMj22O9p4DngoYoVqnoS+EBE7Aa/aTKqyraj20hNT2Xl/pWUlJcwpusYfjzqx1zW8zJr4GACXkOHsCoqLSKvKK9ZRifxV3LqoqpHAFT1iIgkeCnTAzjosZwB9BCRTu7y0yIyAdgD3K+qmSIyHOipqktF5CHq72URKQMW49w+PLtdKCAi9wD3APTq1asBhzetyamSUyzbt4wF6QtIy00jKjSKmwfdzMykmfTrVPe3VWNaqoiQCLpFdWuWc/ksOYnIu4C3J36P1fcQXtYpTsyJwAZVfVBEHgTmuc+nfgPc0cBQv6mqh9zbgYuBbwF/91ZQVV8CXgJISUnxmsBM67Xv+D4Wpi/krd1vkV+Sz8CYgfz0op8yrd802oW283d4xrQqPktOqnp5TdtEJFNEurm1pm6AtwYIGZy59QdOQloH5ACngDfc9YuAu4BonOdJ69yHzl2BJSIyvbbnSap6yP2dLyL/AkZTQ3IybU9peSnvHXyP1PRUNh7ZSEhQCFN6T2FW0iyGJwy3Bg7G+Ii/bustAW4H5rq/3/JSZgXwSxGJcZenAo+qqorIf3AS1xpgMrBdVY8DlR0N3FZ4D9WWmEQkBOikqtkiEgpMA949x/dmWoHswmwW71zMop2LyDyVSdf2Xfnh8B9y48Ab692fxRjTeP5KTnOBhSJyF3AAuBlARFKA76vq3aqaKyJPAx+7+/y8onEETuOJV0VkPnAU+E5dJxSR/UAHIExErsdJdl8BK9zEFIyTmHw6Ff2cxZ9xNL+Y6IgQoiNCq/0OoYOXde3DQggKsm/ovqaqfJL1CQvSFrDqwCpKy0sZ220sPxnzE8Ynjm9xnRiNacmkhmf/pg4pKSm6eXPDW5//KPVTdmUWUFBcSn5RCflFpZSW1/5vIAJRYSFnJa3qya1DDQkvOiKUqPAQgi3BeXWq5BRL9y4lNT2VXXm7iA6N5roB1zEzaSZ9Ovbxd3jGtCoiskVVU+oqZ18Fm9lvZw2vsqyqFJWUk19UwomiMwkrv8rrim1n1h0tKGZv9kkK3PWny86eqK26qPAQosJDvCa3DjUkPM/aXFR4CCHBrWeU5T3H9rAgfQFL9izhZMlJkmOTeXLsk1zV9ypr4GCMn1ly8jMRITIsmMiwYBI6NP44RSVl1RLamdcnvCW84hLyTp3mQO6pynXFpXUnuHZhwR4JLpSqtyLrd6sy1I8JrqS8hLUH1rIgfQGbvt5EaFAoV/S5gplJMxnaeag1cDAmQFhyaiUiQoOJCA2mc3R4o49RXFpWWROrWmPzntycxFfKoWOF5BeVUlBUSmGJ93G9qsYaRFS4t9pa/W9VhocEN+i9ZZ3KYvHOxby+83WyCrPo3r47PxrxI24ceGO9J7gzxjQfS06mUnhIMOFRwcRFNT7BlZSVVya4Ex63JasmtrNvVX59oqhy+6nTdSe4sJAgosPrSG7hweSW72Db8Xf48vgGyrWMEZ0v4ofDHmFK7wm0Cwu1mpIxAcqSk2lSocFBxLQPI6Z94weFLC0r52Rx2dnJrfhMDc7brcrs7JNuuQKKIz8mNOa/BIdnoWWRlBy7mNN5Y3hvRzzvrS8BVhEaLO4tyrOTW223Kj1rc5GhwZbgjPEBS04m4IQEB9GxXRAd2zVs/K5debtYkL6A/+z5D1J6iqSYwVzT+3uMiJ/A6ZIQj2dwXmpz7uuDuacqk19BcSl1NWYNDhIvz+AadquyfZglOGOqs+RkWrSSshJWH1hNanoqWzK3EBYUxpV9r2RW0iwu7HzhOR27vFw5ebrUbfbvvdVk9eTmPIMrIr8ov3JdHT0FCBJoH15Tba1+rSqjrC+caWUsOZkW6euTX/P6ztdZvGsx2YXZ9IjqwYMjH+T6AdcTExFT9wHqIShI3D/+oXTr2LhjqCqnTpdV9murLbl53sbMPFHE7qz694UDKmtwPWPbMTEpgUnJCQzqEmW1MtMiWSfcRmpsJ1zTeKrKpq83kZqWytqDaynXcsYljmNm0kwu6X4JwUENa8HXUlT2hSv23geu+q3KHUdO8OXhEwD06BTJ5MFOorqoXxwRoa3zGpmWwzrhmlYj/3Q+S/YsYUH6AvYd30en8E58+/xvc/Ogm+kZXffsoC1dlb5w0fXb58jxQtamHWVNWiYLNx/k7//9isjQYC4dGM/k5AQmJifQpUOEbwM35hxYzamRrObke+m56aSmp/L23rcpLC1kSPwQZibP5Io+VxAe3Pjm7m1NUUkZ/92bw5odWaxJy+LQsUIALuzRkYnJCUxOTuDCHh3tmZVpFvWtOVlyaiRLTr5RUlbCqq9WkZqeyqdZnxIeHM5Vfa9iVvIszo8739/htXiqSnpmPqvdRPXJgTxUIT4qnEnJnZmU3IVLB8YTFW43VYxvWHLyMUtOTetIwREW7VzE4l2LyS3KpWd0T2YmzeT6AdfTMbyRrRFMnXJPnmZdupOo3tt5lPyiUsKCgxjTL5bJyQlMSu5CrzgbZ9A0HUtOPmbJ6dyVazkbj2wkNS2V9zLeQ1W5LPEyZiXPYmz3sQRJ6xlktiUoKStn8/481qRlsjoti71HTwIwICHKTVQJjOwd06oG/zXNz5KTj1lyarzjxccrGzh8deIrYiNiuXHgjcwYNIMeUT38HZ5x7cs+yZq0LNamZfHRvhxKypQOESFMSEpg8uAELhvUmU7tGj8SiGmbLDn5mCWnhtuRs4PU9FSW7V1GUVkRQzsPZVbyLKb2nkpYsP2RC2T5RSV8sCub1W6yyjl5miCBkb1jmJTchcmDExiYYH2qTN0sOfmYJaf6KS4rZuX+laSmp/LZ0c+ICI7gmn7XMDNpJoPjBvs7PNMI5eXKtoxjrEnLYvWOLLYfcfpUJcZEOrf/BndhTN9Y61NlvLLk5GOWnGp3qOAQi9IX8e9d/yavOI8+HfowM2km0wdMp0PYOUxcZQKOZ5+qD3ZnU1RSTruwYC4dEM8k91lVgvWpMi5LTj5myels5VrOh4c/JDUtlfUZ6xERJiROYFbyLMZ0G2MNHNqAopIy/rsnh9VpmazZkcXh40WA06dqUrLzrOqC7tanqi2z5ORjlpzOOF58nDd3v8mC9AUczD9IbEQsNw28iVuSbqFr+67+Ds/4iaqS9nU+a9Kq9qnqHB3OpKQEJg1O4NIB8bS3PlVtiiUnH7PkBF9mf0lqeirv7HuH4rJiRiSMYGbSTKb0nkJocMOmuzCtX05BMe/tPMrqtCzWpx8lv7hqn6rJg7vQM9b6VLV2lpx8rK0mp6LSIlbsX8GC9AV8nv05kSGRTOs3jZlJM0mKTfJ3eKaFKCkr5+P9uc6QSuln+lQNTIhi0uAEJid3YUSvTtanqhWy5ORjbS05Hcw/6DRw2P1vjhcfp2/Hvk4Dh/7TiQ6r52ikxtSgok/VmrRMPtqbS2m50jEylAlJnZmUbH2qWhNLTj7WFpJTWXkZGw5vIDUtlQ8OfUCQBDGp1yRmJs1kdNfR1qfF+MSJij5VO7JYl36mT1VK71i3VpXAAOtT1WJZcvKx1pyc8oryeGP3GyxMX8ihgkPER8YzY9AMbhp4kzVwMM2qzO1TtbZan6qesZFMTu7CpOQExvSLJTzE+lS1FJacfKy1JSdV5fPsz1mQvoDl+5Zzuvw0KV1SmJk8k8m9JhMaZA0cjP8dPlbI2vQs1uzI4oPd2RSXOn2qxg10+lRNTLI+VYHOkpOPtZbkVFhayPJ9y0lNT2V7znbahbTj2v7XMjNpJgNjBvo7PGNqVHi6jP/uzXaeVXn0qRqS6PapSu7C+d07WJ+qAGPJycdaenI6cOIAC9IX8ObuNzlx+gT9O/ZnVvIsru1/Le1D2/s7PGMaxLNP1eodmXx68BiqkBAdzkTrUxVQLDn5WEtMTmXlZazPWM+C9AVsOLyBEAlhcu/JzEyaSUqXFHvAbFqNnIJi1qUfZU1aFut3nulTdVH/uMrpP6xPlX9YcvKxlpSccgpzKhs4HDl5hITIBGYkOQ0cEtol+Ds8Y3yqSp+qtCz2Zjt9qgZ1iaocUX14T+tT1VwsOflYoCcnVWXb0W2kpqeycv9KSspLGNN1DDOTZzKh5wRr4GDarL1HCyqHVNq07+w+VRMGJdCxnf3/8BVLTj4WqMnpVMkplu1bxoL0BaTlphEVGsX0/tOZmTSTfp36+Ts8YwKKZ5+qtelZ5J48TXCQMLJ3jDukUgL9O1ufqqZkycnHAi057T++nwXpC3hr91vkl+QzMGYgs5JmMa3fNNqF2r11Y+pS0adqzY4sVqdlscPtU9Urtl3l1B/Wp+rcWXLysUBITqXlpbyX8R6paalsPLKRkKAQpvSewqykWQxPGG7f9ow5B4ePFVZOU1+9T9Xk5C5MSO5MQrT1qWqogE5OIhILLAD6APuBW1Q1z0u524HH3cVnVPUVd30Y8CIwASgHHlPVxR77zQAWAaNUdbOITAHmAmHAaeBhVV3jlh0J/A2IBJYBP9J6XBR/JqfswmwW71zMop2LyDyVSZd2Xbgl6RZuHHgj8ZHxfonJmNasok/VardRxRG3T9XQxI6VjSrO797BvhDWQ6Anp+eAXFWdKyJzgBhVfaRamVhgM5ACKLAFGKmqeSLyFBCsqo+LSBAQq6rZ7n7RwNs4ieh+NzkNBzJV9bCIXACsUNUebvlNwI+AjTjJ6QVVfaeu99DcyUlV+TTrU1LTUll1YBWl5aVc1O0iZiXP4rLEywgJsv4bxjQHVWXHkXzWpGWyJi2rsk9Vlw5un6rkBC4dGE+7MPs/6U2gJ6d0YIKqHhGRbsA6VU2qVuZWt8z33OU/uuVeE5GDQLKqnvRy7PnAu8BDwEOqurnadgGyge5ALLBWVZO9nbM2zZWcTpWcYunepSxIX8DOvJ1Eh0Zz3YDruCXpFvp27Ovz8xtjaue1T1VIEGP7xTF5sDOkkvWpOqO+yclfqb2Lqh4BcBOUt842PYCDHssZQA8R6eQuPy0iE4A9ODWkTLeG1FNVl4rIQzWc+ybgU1UtFpEe7nGrnKPxb6vp7D22l9T0VJbsWcLJkpMkxybz5NgnuarvVdbAwZgAEhcVzk0jE7lpZCKnS8vZvD+X1W5T9Z+99SXwpfWpagSfJScReRfwNoT1Y/U9hJd1ihNzIrBBVR8UkQeBee7zqd8Ad9QS0/nAs8DUOs5R0/73APcA9OrVqx5voWFKyktYd3AdqWmpbPp6E6FBoVzR5wpmJs1kaOehdj/bmAAXFhLExQPiuXhAPD+ddl6VPlV/fn8vf3hvD53ahTJhUGcmDe7CZQM7W5+qGvgsOanq5TVtE5FMEenmcVsvy0uxDJwGDxUSgXVADnAKeMNdvwi4C4gGLgDWuX/EuwJLRGS6+9wp0d3n26q6x+McidXOcbiW9/QS8BI4t/VqKtdQR08d5fVdr/N6+utkFWbRrX03fjTiR9ww4AbiIuOa6jTGmGbWr3MU/TpHcfe4fpwoKuH9ndmsTstkXfpR3tx6mOAgIaV3DJMHO8+qrE/VGf565vQrIMejQUSsqv64WplYnEYQI9xVn+A0iMgVkVTgJVVdIyJ3ANeo6s3V9l+H+8zJvRX4HvBzz1Z9brmPgR8CH+E0iPh/qrqsrvdwrs+cVJXNmZtZkL6A1V+tplRLuaT7JcxKnsW4HuMIDrK+FMa0VnX1qZo8OIHRfVtnn6pAbxARBywEegEHgJvdpJMCfF9V73bL3Qn8xN3tF6r6sru+N/Aq0Ak4CnxHVQ9UO8c6ziSnx4FHgV0eRaaqapZ7zr/hNCV/B/ihL5uSF5wuqGzgsPvYbjqEdeD6AdczM2kmvTo0/a1CY0zgO3SskLXu7b8Nbp+q9mHBjBvYmUluo4rO0eH+DrNJBHRyag0ak5zKtZyr/301hwoOcV7cecxKmsWVfa8kMiTSR1EaY1qawtNlfLgnm9VuB+DKPlU9OzEpKaHF96my5ORjja05Ld+3nB5RPbgg/oIW++EyxjQPzz5Vq9Oy2OrRp8oZUqkLlwyIa1F9qiw5+VggDF9kjGlbsiv7VGWyfmc2BW6fqovdeaomJieQGBPYXU0sOfmYJSdjjD+dLnXnqXJn/92fcwqApC7RTBqcwOTkBIb3iiE4wKapt+TkY5acjDGBpKJP1eodWXy835mnKqZdKBPcIZXGD+pMx0j/96my5ORjlpyMMYHqeGEJ7+9yhlRal360cp6qM32qutC/c3u/PPe25ORjlpyMMS1BWbmy9eAxp1HFjizSvs4HoHec26cquQuj+8YSFtI8QypZcvIxS07GmJbokDtP1ZodmXy4J4fi0nKiwkMYNzCeicm+71NlycnHLDkZY1o6zz5Va3Zk8fWJM32qJruz/zZ1nypLTj5myckY05qoKtuPnKgcUmlbhmefqi5MTk7gkgHxRIad25BKlpx8zJKTMaY1q61P1fO3DCO2fVijjhvo8zkZY4wJYPFR4cwYmcgMd56qj/fnsnpHFlsP5tGpGZqkW3IyxhhTq7CQIC4ZEM8lA+Kb7Zw2HaMxxpiAY8nJGGNMwLHkZIwxJuBYcjLGGBNwLDkZY4wJOJacjDHGBBxLTsYYYwKOJSdjjDEBx4YvaiQROQp81cjd44HsJgynqVhcDWNxNYzF1TCtMa5sAFW9sq6Clpz8QEQ212dsqeZmcTWMxdUwFlfDtPW47LaeMcaYgGPJyRhjTMCx5OQfL/k7gBpYXA1jcTWMxdUwbToue+ZkjDEm4FjNyRhjTMCx5NTERCRYRD4VkaVetoWLyAIR2S0iH4lIH49tj7rr00XkimaO60ER2S4in4nIahHp7bGtTES2uj9LmjmuO0TkqMf57/bYdruI7HJ/bm/muH7jEdNOETnmsc3X12u/iHzuHv+sqZjF8YL7WfpMREZ4bPPZNatHXN904/lMRD4UkaH13dfHcU0QkeMe/2Y/89h2pfv/cbeIzGnmuB72iOkL93MVW599zzGuTiLyuoikicgOERlbbXvzfb5U1X6a8Ad4EPgXsNTLtnuBP7ivZwEL3NfnAduAcKAvsAcIbsa4JgLt3Nc/qIjLXS7w4/W6A3jRy/pYYK/7O8Z9HdNccVUr90Pgr814vfYD8bVsvxp4BxDgIuCj5rhm9Yjr4orzAVdVxFWffX0c14QaPnvB7v/DfkCY+//zvOaKq1rZa4E1zXS9XgHudl+HAZ389fmymlMTEpFE4BrgzzUUuQ7nHx/gdWCyiIi7PlVVi1V1H7AbGN1ccanqWlU95S5uBBKb6tznElctrgBWqWququYBq4A6O/X5KK5bgdea6txN4Drg7+rYCHQSkW74+JrVRVU/dM8LzfgZOwejgd2quldVTwOpONfWH5rlMyYiHYDxwF8AVPW0qh6rVqzZPl+WnJrWfODHQHkN23sABwFUtRQ4DsR5rndluOuaKy5Pd+F8M6oQISKbRWSjiFzfhDHVN66b3NsHr4tIT3ddQFwv9/ZnX2CNx2pfXi8ABVaKyBYRucfL9pquja+vWV1xear+GWvIvr6Ia6yIbBORd0TkfHddQFwvEWmH80d+cUP3bYR+wFHgZfeW9p9FpH21Ms32+Qo5l53NGSIyDchS1S0iMqGmYl7WaS3rmyuuirK3ASnAZR6re6nqYRHpB6wRkc9VdU8zxfUf4DVVLRaR7+PUOicRINcL59bs66pa5rHOJ9fLwyXu8ROAVSKSpqrrPd+Cl318+hmrZ1xOcCITcZLTpQ3d10dxfQL0VtUCEbkaeBMYSIBcL5xbehtUNbcR+zZUCDAC+KGqfiQivwXmAD/1KNNsny+rOTWdS4DpIrIf5xbAJBH5R7UyGUBPABEJAToCuZ7rXYnA4WaMCxG5HHgMmK6qxRXrVfWw+3svsA4Y3lxxqWqORyx/Aka6r/1+vVyzqHa7xYfXq/rxs4A3OPv2b03XxpfXrD5xISJDcG6VXqeqOQ3Z11dxqeoJVS1wXy8DQkUkngC4Xq7aPmNNfb0ygAxV/chdfh0nWVUv0zyfL188VGvrP9T8kPU+qjaIWOi+Pp+qDSL20sQNIuqIazjOw9+B1dbHAOHu63hgF034ULgecXXzeH0DsNF9HQvsc+OLcV/HNldc7rYknAfT0lzXC2gPRHu8/hC4slqZa6j6wHqTr69ZPePqhfMs9eKG7uvjuLpW/Bvi/JE/4F67EPf/YV/ONIg4v7nicrdVfHlt3xzXyz3m+0CS+/pJ4Ff++nzZbT0fE5GfA5tVdQnOg8ZXRWQ3zoduFoCqfikiC4HtQClwn1a9VeTruH4FRAGLnPYZHFDV6cBg4I8iUo5Ty56rqtubMa7ZIjId55rk4rTeQ1VzReRp4GN3t59r1dsevo4LnIfUqer+z3T5+np1Ad5w/41CgH+p6nL3lieq+gdgGU6Lqt3AKeA77jZfXrP6xPUznOerv3PLlaozeKjXfZsxrhnAD0SkFCgEZrn/pqUicj+wAqfl3l9V9ctmjAucL2QrVfVkXfs2UVzgtD79p4iE4STn7/jr82UjRBhjjAk49szJGGNMwLHkZIwxJuBYcjLGGBNwLDkZY4wJOJacjDHGBBxLTsYYYwKOJSdjWjFxpleIb+S+d4hI96Y4ljENZcnJGFOTO4DudRUyxhcsORnTDESkjzgTuP1ZnMnj/ikil4vIBndyttHuz4fuiNAfikiSu++DIvJX9/WF7v7tajhPnIisdI/xRzwG5BSR20RkkziT1P1RRILd9QUi8msR+UScySY7i8gMnEGA/+mWj3QP80O33OcikuzLa2baNktOxjSfAcBvgSFAMvANnNG5HwJ+AqQB41V1OM5wP79095sPDBCRG4CXge/pmfm3qnsC+MA9xhKcMe0QkcHATJwRrYcBZcA33X3aA5+o6gjgPeAJVX0d2Ax8U1WHqWqhWzbbLfd7N25jfMLG1jOm+exT1c8BRORLYLWqqoh8DvTBGejzFREZiDPdQCiAqpaLyB3AZ8AfVXVDLecYD9zo7ve2iFRM8DcZZ1T3j91x2SKBLHdbObDAff0P4N+1HL9i25aK8xjjC5acjGk+xR6vyz2Wy3H+Lz4NrFXVG0SkD86UGxUGAgXU7xmQtwEzBXhFVR9t5P4VKmIuw/5+GB+y23rGBI6OwCH39R0VK0WkI87twPFAnPs8qCbrcW/XichVONMXAKwGZrgT1CEiseLM5AvO34GKY34D+MB9nQ9En8P7MabRLDkZEzieA/5XRDbgTNNQ4TfA71R1J84ssnMrkowXTwHjReQTYCrO/ES4U3c8jjO992fAKqCbu89J4HwR2YIz0/DP3fV/A/5QrUGEMc3Cpswwpo0TkQJVjfJ3HMZ4spqTMcaYgGM1J2NaIBH5DvCjaqs3qOp9/ojHmKZmyckYY0zAsdt6xhhjAo4lJ2OMMQHHkpMxxpiAY8nJGGNMwLHkZIwxJuD8/w3lgF1DPpDnAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize results\n",
    "print(\"Best: %f using %s\" % (gsearch2_2.best_score_, gsearch2_2.best_params_))\n",
    "test_means = gsearch2_2.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = gsearch2_2.cv_results_[ 'std_test_score' ]\n",
    "train_means = gsearch2_2.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = gsearch2_2.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "pd.DataFrame(gsearch2_2.cv_results_).to_csv('my_preds_maxdepth_min_child_weights_2.csv')\n",
    "\n",
    "# plot results\n",
    "test_scores = np.array(test_means).reshape(len(min_child_weight), len(max_depth))\n",
    "train_scores = np.array(train_means).reshape(len(min_child_weight), len(max_depth))\n",
    "\n",
    "for i, value in enumerate(min_child_weight):\n",
    "    pyplot.plot(max_depth, test_scores[i], label= 'test_min_child_weight:'   + str(value))\n",
    "#for i, value in enumerate(min_child_weight):\n",
    "#    pyplot.plot(max_depth, train_scores[i], label= 'train_min_child_weight:'   + str(value))\n",
    "    \n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'max_depth' )                                                                                                      \n",
    "pyplot.ylabel( '- Log Loss' )\n",
    "pyplot.savefig( 'max_depth_vs_min_child_weght2.png' )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ice2018/anaconda2/lib/python2.7/site-packages/sklearn/model_selection/_search.py:762: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: -0.06399, std: 0.00113, params: {'gamma': 0.0},\n",
       "  mean: -0.06413, std: 0.00112, params: {'gamma': 0.1},\n",
       "  mean: -0.06407, std: 0.00114, params: {'gamma': 0.2},\n",
       "  mean: -0.06413, std: 0.00111, params: {'gamma': 0.3},\n",
       "  mean: -0.06406, std: 0.00119, params: {'gamma': 0.4}],\n",
       " {'gamma': 0.0},\n",
       " -0.06398936452777433)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Grid seach on subsample and max_features\n",
    "#Choose all predictors except target & IDcols\n",
    "param_test3 = {\n",
    "    'gamma':[i/10.0 for i in range(0,5)]\n",
    "}\n",
    "\n",
    "xgb2_3 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=93,  #第一轮参数调整得到的n_estimators最优值\n",
    "        max_depth=5,\n",
    "        min_child_weight=2,\n",
    "        gamma=0,\n",
    "        subsample=0.3,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel = 0.7,\n",
    "        objective= 'binary:logistic',\n",
    "        seed=3)\n",
    "\n",
    "\n",
    "gsearch2_3 = GridSearchCV(xgb2_3, param_grid = param_test3, scoring='neg_log_loss',n_jobs=-1, cv=kfold)\n",
    "gsearch2_3.fit(X_train , y_train)\n",
    "\n",
    "gsearch2_3.grid_scores_, gsearch2_3.best_params_,     gsearch2_3.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
